2024-09-16T10:38:14.399 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T10:38:14.419 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:38:14.513 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688 branch: reef description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/force-sync-many workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906688' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T10:38:14.513 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T10:38:14.514 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T10:38:14.514 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T10:38:14.516 INFO:teuthology.task.internal:Checking packages... 2024-09-16T10:38:14.539 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T10:38:14.539 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T10:38:14.539 INFO:teuthology.packaging:ref: None 2024-09-16T10:38:14.540 INFO:teuthology.packaging:tag: None 2024-09-16T10:38:14.540 INFO:teuthology.packaging:branch: reef 2024-09-16T10:38:14.540 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:38:14.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-16T10:38:14.807 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1focal 2024-09-16T10:38:14.809 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T10:38:14.815 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T10:38:14.815 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T10:38:14.822 INFO:teuthology.task.internal:Saving configuration 2024-09-16T10:38:14.836 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T10:38:14.843 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T10:38:14.865 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:32:30.204607', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T10:38:14.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:32:30.202720', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T10:38:14.888 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T10:38:14.896 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T10:38:14.896 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T10:38:14.896 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T10:38:14.952 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f953fd672e0>, signals=[15]) 2024-09-16T10:38:14.952 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T10:38:14.958 INFO:teuthology.task.internal:Opening connections... 2024-09-16T10:38:14.959 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T10:38:14.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:38:15.048 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T10:38:15.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:38:15.142 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T10:38:15.149 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-09-16T10:38:15.155 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-09-16T10:38:15.156 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-09-16T10:38:15.204 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="Ubuntu" 2024-09-16T10:38:15.204 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T10:38:15.204 INFO:teuthology.orchestra.run.smithi005.stdout:ID=ubuntu 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE=debian 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="20.04" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_CODENAME=focal 2024-09-16T10:38:15.205 INFO:teuthology.orchestra.run.smithi005.stdout:UBUNTU_CODENAME=focal 2024-09-16T10:38:15.206 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-09-16T10:38:15.224 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-09-16T10:38:15.229 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-09-16T10:38:15.229 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-09-16T10:38:15.277 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2024-09-16T10:38:15.277 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T10:38:15.277 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="20.04" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=focal 2024-09-16T10:38:15.278 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=focal 2024-09-16T10:38:15.278 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-09-16T10:38:15.301 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T10:38:15.309 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T10:38:15.316 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T10:38:15.316 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:38:15.318 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:38:15.324 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T10:38:15.330 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T10:38:15.330 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:38:15.364 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:38:15.382 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T10:38:15.432 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T10:38:15.446 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T10:38:15.446 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T10:38:15.447 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:38:15.447 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:38:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:38:15.453 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-16T10:38:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:38:15.454 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-16T10:38:15.459 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-124-generic 2024-09-16T10:38:15.460 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.4.0-124-generic 2024-09-16T10:38:15.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-16T10:38:15.461 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-124-generic 2024-09-16T10:38:15.461 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.4.0-124-generic 2024-09-16T10:38:15.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-09-16T10:38:15.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T10:38:15.617 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-16T10:38:15.729 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:38:15.741 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T10:38:15.788 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:38:15.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T10:38:15.788 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T10:38:15.806 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T10:38:15.841 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T10:38:15.850 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T10:38:16.119 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T10:38:16.190 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T10:38:16.273 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T10:38:16.278 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T10:38:16.292 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T10:38:16.297 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T10:38:16.297 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T10:38:16.320 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T10:38:16.321 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T10:38:16.325 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T10:38:16.325 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T10:38:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T10:38:16.335 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T10:38:16.337 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T10:38:16.341 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T10:38:16.344 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T10:38:16.345 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T10:38:16.346 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T10:38:16.346 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T10:38:16.362 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T10:38:16.362 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T10:38:16.379 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T10:38:16.379 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T10:38:16.382 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T10:38:16.382 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T10:38:16.387 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T10:38:16.387 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T10:38:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T10:38:16.399 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T10:38:16.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T10:38:16.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T10:38:16.402 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T10:38:16.443 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T10:38:16.453 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T10:38:16.458 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T10:38:16.458 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T10:38:16.459 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T10:38:16.459 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T10:38:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T10:38:16.461 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T10:38:16.489 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T10:38:16.492 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T10:38:16.492 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T10:38:16.497 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T10:38:16.505 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T10:38:16.507 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T10:38:16.508 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T10:38:16.508 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T10:38:16.508 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T10:38:16.508 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T10:38:16.523 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T10:38:16.524 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T10:38:16.525 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T10:38:16.525 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T10:38:16.540 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T10:38:16.541 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T10:38:16.794 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T10:38:16.986 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T10:38:17.000 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T10:38:17.000 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T10:38:17.090 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T10:38:17.095 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T10:38:17.105 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T10:38:17.105 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T10:38:17.134 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T10:38:17.154 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T10:38:17.161 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T10:38:17.162 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T10:38:17.162 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T10:38:17.235 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T10:38:17.235 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T10:38:20.620 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 21.8 MB in 3s (7,046 kB/s) 2024-09-16T10:38:20.867 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 21.8 MB in 3s (6,876 kB/s) 2024-09-16T10:38:22.195 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:38:22.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:38:22.299 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:38:22.419 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:38:22.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:38:22.518 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T10:38:22.519 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T10:38:22.524 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:38:22.727 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-09-16T10:38:22.728 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T10:38:22.728 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T10:38:22.728 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T10:38:22.729 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-09-16T10:38:22.729 INFO:teuthology.orchestra.run.smithi143.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T10:38:22.729 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-09-16T10:38:22.729 INFO:teuthology.orchestra.run.smithi143.stdout: thermald 2024-09-16T10:38:22.746 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-16T10:38:22.747 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-16T10:38:22.788 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-09-16T10:38:22.789 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T10:38:22.789 INFO:teuthology.orchestra.run.smithi143.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T10:38:22.789 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T10:38:22.790 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2024-09-16T10:38:22.790 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T10:38:22.955 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-16T10:38:22.955 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T10:38:22.955 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T10:38:22.955 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T10:38:22.956 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-16T10:38:22.956 INFO:teuthology.orchestra.run.smithi005.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T10:38:22.956 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-16T10:38:22.957 INFO:teuthology.orchestra.run.smithi005.stdout: thermald 2024-09-16T10:38:22.993 INFO:teuthology.orchestra.run.smithi143.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T10:38:22.993 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 77.2 MB of archives. 2024-09-16T10:38:22.993 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T10:38:22.994 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T10:38:23.015 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-16T10:38:23.015 INFO:teuthology.orchestra.run.smithi005.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T10:38:23.015 INFO:teuthology.orchestra.run.smithi005.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T10:38:23.015 INFO:teuthology.orchestra.run.smithi005.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T10:38:23.017 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-09-16T10:38:23.017 INFO:teuthology.orchestra.run.smithi005.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T10:38:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T10:38:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 77.2 MB of archives. 2024-09-16T10:38:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T10:38:23.199 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T10:38:24.169 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T10:38:24.368 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T10:38:24.443 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T10:38:24.687 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T10:38:25.765 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T10:38:25.765 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T10:38:25.766 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T10:38:26.006 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T10:38:26.007 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T10:38:26.007 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T10:38:26.096 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T10:38:26.116 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T10:38:26.283 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T10:38:26.323 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T10:38:26.825 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 77.2 MB in 3s (23.3 MB/s) 2024-09-16T10:38:26.996 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 77.2 MB in 3s (23.4 MB/s) 2024-09-16T10:38:27.009 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T10:38:27.179 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T10:38:28.791 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T10:38:28.796 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:28.877 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T10:38:28.882 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:28.914 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:28.952 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:30.872 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T10:38:30.885 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:30.902 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T10:38:30.916 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:30.989 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:31.007 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:32.082 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T10:38:32.098 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:32.108 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T10:38:32.121 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:32.138 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:32.166 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:36.744 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:36.868 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:36.942 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:37.067 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:37.185 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:37.315 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:37.411 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:37.537 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:37.650 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T10:38:37.665 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T10:38:37.733 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T10:38:37.881 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T10:38:37.896 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T10:38:37.964 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T10:38:43.098 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T10:38:43.122 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:43.147 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T10:38:43.164 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:43.169 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T10:38:43.213 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:45.902 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:45.927 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T10:38:46.040 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:46.085 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T10:38:46.393 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:46.469 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:47.597 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:47.698 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:48.902 INFO:teuthology.orchestra.run.smithi005.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:38:48.902 INFO:teuthology.orchestra.run.smithi005.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T10:38:49.059 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:49.146 INFO:teuthology.orchestra.run.smithi143.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:38:49.146 INFO:teuthology.orchestra.run.smithi143.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T10:38:49.283 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:50.285 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T10:38:50.436 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T10:38:50.468 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T10:38:50.562 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:50.619 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T10:38:50.688 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T10:38:50.745 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:50.789 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T10:38:50.871 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T10:38:50.916 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:50.997 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T10:38:51.123 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T10:38:51.222 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T10:38:51.222 INFO:teuthology.orchestra.run.smithi143.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:38:51.243 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T10:38:51.243 INFO:teuthology.orchestra.run.smithi005.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:39:12.878 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T10:39:12.878 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub' 2024-09-16T10:39:12.886 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:39:12.897 INFO:teuthology.orchestra.run.smithi143.stdout:Generating grub configuration file ... 2024-09-16T10:39:13.211 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:39:13.229 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:39:13.591 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T10:39:13.603 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T10:39:14.531 INFO:teuthology.orchestra.run.smithi005.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T10:39:14.532 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub' 2024-09-16T10:39:14.550 INFO:teuthology.orchestra.run.smithi005.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:39:14.576 INFO:teuthology.orchestra.run.smithi005.stdout:Generating grub configuration file ... 2024-09-16T10:39:14.872 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:39:14.892 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:39:15.246 INFO:teuthology.orchestra.run.smithi005.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T10:39:15.258 INFO:teuthology.orchestra.run.smithi005.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T10:39:15.429 INFO:teuthology.orchestra.run.smithi143.stdout:done 2024-09-16T10:39:15.850 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-09-16T10:39:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-09-16T10:39:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-09-16T10:39:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-09-16T10:39:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-09-16T10:39:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 20 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.4.0.195.193 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:39:15.894 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-09-16T10:39:15.894 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T10:39:15.895 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:39:15.895 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:39:15.895 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T10:39:15.895 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:39:15.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-09-16T10:39:16.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T10:39:16.145 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T10:39:16.260 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:39:16.342 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T10:39:16.425 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T10:39:17.312 INFO:teuthology.orchestra.run.smithi005.stdout:done 2024-09-16T10:39:17.703 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 20 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:39:17.758 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.195.193 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:39:17.759 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-16T10:39:17.759 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T10:39:17.760 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:39:17.760 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:39:17.760 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T10:39:17.760 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:39:17.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get clean 2024-09-16T10:39:17.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-16T10:39:17.968 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:39:17.975 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T10:39:17.992 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T10:39:18.011 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:39:18.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:39:18.085 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T10:39:18.111 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:39:18.330 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T10:39:18.331 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T10:39:18.568 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T10:39:18.569 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T10:39:18.571 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-09-16T10:39:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-09-16T10:39:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-09-16T10:39:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-09-16T10:39:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-09-16T10:39:18.593 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 20 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.4.0.195.193 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:39:18.594 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-09-16T10:39:18.595 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-16T10:39:18.642 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.FhJeXpoZZo 2024-09-16T10:39:18.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FhJeXpoZZo 2024-09-16T10:39:18.658 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.FhJeXpoZZo 2024-09-16T10:39:18.740 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.FhJeXpoZZo is 9KB 2024-09-16T10:39:18.756 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.FhJeXpoZZo 2024-09-16T10:39:18.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T10:39:18.819 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:39:18.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T10:39:18.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T10:39:18.855 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T10:39:18.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo update-grub 2024-09-16T10:39:19.882 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:39:19.910 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:39:19.985 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:39:20.172 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub' 2024-09-16T10:39:20.182 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:39:20.191 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-09-16T10:39:20.192 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-16T10:39:20.193 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-16T10:39:20.433 INFO:teuthology.orchestra.run.smithi005.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T10:39:20.433 INFO:teuthology.orchestra.run.smithi005.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T10:39:20.435 DEBUG:teuthology.orchestra.run.smithi005:> dpkg -s linux-image-generic 2024-09-16T10:39:20.441 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:39:20.454 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Package: linux-image-generic 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Status: install ok installed 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Priority: optional 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Section: kernel 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Installed-Size: 20 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: amd64 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Source: linux-meta 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Version: 5.4.0.195.193 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:39:20.458 INFO:teuthology.orchestra.run.smithi005.stdout:Recommends: thermald 2024-09-16T10:39:20.459 INFO:teuthology.orchestra.run.smithi005.stdout:Description: Generic Linux kernel image 2024-09-16T10:39:20.459 INFO:teuthology.orchestra.run.smithi005.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:39:20.459 INFO:teuthology.orchestra.run.smithi005.stdout: available. 2024-09-16T10:39:20.459 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-16T10:39:20.465 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.xz1B67EeRk 2024-09-16T10:39:20.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xz1B67EeRk 2024-09-16T10:39:20.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.xz1B67EeRk 2024-09-16T10:39:20.566 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.xz1B67EeRk is 9KB 2024-09-16T10:39:20.581 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.xz1B67EeRk 2024-09-16T10:39:20.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T10:39:20.650 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:39:20.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T10:39:20.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T10:39:20.685 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T10:39:20.685 DEBUG:teuthology.orchestra.run.smithi005:> sudo update-grub 2024-09-16T10:39:20.837 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T10:39:20.849 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T10:39:21.637 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-09-16T10:39:21.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-09-16T10:39:21.937 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub' 2024-09-16T10:39:21.949 INFO:teuthology.orchestra.run.smithi005.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:39:21.958 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-09-16T10:39:22.199 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T10:39:22.211 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T10:39:22.573 INFO:teuthology.orchestra.run.smithi005.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T10:39:22.585 INFO:teuthology.orchestra.run.smithi005.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T10:39:23.710 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-09-16T10:39:23.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-09-16T10:39:51.669 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T10:39:51.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:39:51.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:39:53.717 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T10:39:53.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:39:53.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:40:10.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-09-16T10:40:12.097 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-09-16T10:40:19.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:40:19.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:40:21.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:40:21.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:40:22.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-09-16T10:40:34.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:40:34.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:41:05.845 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-16T10:41:06.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:41:06.421 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-16T10:41:06.421 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-16T10:41:06.428 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-195-generic 2024-09-16T10:41:06.428 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T10:41:06.428 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:41:06.428 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T10:41:07.430 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T10:41:07.430 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:41:07.517 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-09-16T10:41:07.541 DEBUG:teuthology.parallel:result is None 2024-09-16T10:41:21.102 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T10:41:33.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:41:33.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:41:33.360 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-16T10:41:34.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:41:34.002 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-16T10:41:34.003 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-16T10:41:34.009 INFO:teuthology.orchestra.run.smithi005.stdout:5.4.0-195-generic 2024-09-16T10:41:34.009 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T10:41:34.009 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:41:34.009 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T10:41:35.010 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T10:41:35.011 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:41:35.104 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-09-16T10:41:35.145 DEBUG:teuthology.parallel:result is None 2024-09-16T10:41:35.145 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T10:41:35.156 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T10:41:35.156 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:41:35.158 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:41:35.166 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T10:41:35.172 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T10:41:35.178 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T10:41:35.178 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:41:35.180 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:41:35.203 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T10:41:35.209 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T10:41:35.210 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:41:35.212 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:41:35.228 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:41:35.237 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:41:35.264 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:41:35.274 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:41:35.276 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T10:41:35.283 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T10:41:35.284 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:41:35.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:41:35.300 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T10:41:35.311 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T10:41:35.311 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:41:35.314 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:41:35.320 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:41:35.365 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:41:35.370 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:41:35.371 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:41:35.395 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:41:35.402 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:41:35.410 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:41:35.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:41:35.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-09-16T10:41:35.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-09-16T10:41:35.525 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T10:41:35.532 INFO:teuthology.task.internal:Starting timer... 2024-09-16T10:41:35.533 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T10:41:35.543 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T10:41:35.552 DEBUG:teuthology.task.selinux:Excluding smithi005: OS 'ubuntu' does not support SELinux 2024-09-16T10:41:35.552 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2024-09-16T10:41:35.552 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T10:41:35.552 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T10:41:35.552 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T10:41:35.552 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T10:41:35.561 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T10:41:35.572 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T10:41:35.573 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T10:47:16.176 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2024-09-16T10:47:16.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:47:16.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:47:16.274 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-16T10:47:16.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-16T10:47:16.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:47:16.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:47:16.486 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-16T10:47:16.611 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T10:47:16.611 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T10:47:16.622 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T10:47:16.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:47:16.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:47:16.626 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:47:16.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:47:16.658 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T10:47:16.659 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Command line: ntpd -gq 2024-09-16T10:47:16.660 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: proto: precision = 0.100 usec (-23) 2024-09-16T10:47:16.660 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:47:16.660 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T10:47:16.660 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Command line: ntpd -gq 2024-09-16T10:47:16.661 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:47:16.661 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2024-09-16T10:47:16.662 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen normally on 4 lo [::1]:123 2024-09-16T10:47:16.663 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2024-09-16T10:47:16.663 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:16 ntpd[13755]: Listening on routing socket on fd #22 for interface updates 2024-09-16T10:47:16.663 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: proto: precision = 0.293 usec (-22) 2024-09-16T10:47:16.664 INFO:teuthology.orchestra.run.smithi005.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:47:16.664 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:47:16.664 INFO:teuthology.orchestra.run.smithi005.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:47:16.665 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:47:16.665 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen normally on 3 ens1f0 172.21.15.5:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen normally on 4 lo [::1]:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1442%3]:123 2024-09-16T10:47:16.666 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:16 ntpd[13647]: Listening on routing socket on fd #22 for interface updates 2024-09-16T10:47:17.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:17.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:17.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:17.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:17.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:17.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:17 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:17.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:17 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:18 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:18.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:18 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:18.660 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:18 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:18.660 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:18 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:19.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:19.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:19.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:19.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:19.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:19.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:19 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:19.661 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:19 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:20.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:20.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:20 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:20.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:20 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:21.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:21.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:21 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:21.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:21 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:22.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:22.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:22.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:22 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:22.665 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:22.666 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:22 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:23.659 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:23.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:23.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:23.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:23.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:23.660 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:23.661 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 10:47:23 ntpd[13755]: ntpd: time slew -0.000619 s 2024-09-16T10:47:23.661 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew -0.000619s 2024-09-16T10:47:23.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T10:47:23.661 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 10:47:23 ntpd[13755]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T10:47:23.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:23.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:23.662 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stdout:16 Sep 10:47:23 ntpd[13647]: ntpd: time slew +0.000199 s 2024-09-16T10:47:23.663 INFO:teuthology.orchestra.run.smithi005.stdout:ntpd: time slew +0.000199s 2024-09-16T10:47:23.664 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T10:47:23.664 INFO:teuthology.orchestra.run.smithi005.stderr:16 Sep 10:47:23 ntpd[13647]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T10:47:23.731 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T10:47:23.732 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-09-16T10:47:23.732 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.732 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.732 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.732 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.741 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T10:47:23.741 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-16T10:47:23.741 INFO:teuthology.orchestra.run.smithi005.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.742 INFO:teuthology.orchestra.run.smithi005.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.742 INFO:teuthology.orchestra.run.smithi005.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.742 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T10:47:23.743 INFO:teuthology.run_tasks:Running task install... 2024-09-16T10:47:23.751 DEBUG:teuthology.task.install:project ceph 2024-09-16T10:47:23.751 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T10:47:23.751 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T10:47:23.751 INFO:teuthology.task.install:Using flavor: default 2024-09-16T10:47:23.757 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T10:47:23.757 INFO:teuthology.task.install:extra packages: [] 2024-09-16T10:47:23.758 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-key list | grep Ceph 2024-09-16T10:47:23.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2024-09-16T10:47:23.869 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T10:47:23.892 INFO:teuthology.orchestra.run.smithi005.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T10:47:23.929 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T10:47:23.929 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T10:47:23.930 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T10:47:23.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:47:23.956 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T10:47:23.956 INFO:teuthology.orchestra.run.smithi005.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T10:47:23.957 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T10:47:23.957 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:47:24.084 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T10:47:24.084 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T10:47:24.101 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T10:47:24.102 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T10:47:24.198 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:47:24.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T10:47:24.212 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:47:24.212 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T10:47:24.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T10:47:24.229 DEBUG:teuthology.orchestra.run.smithi005:> sudo apt-get update 2024-09-16T10:47:24.356 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T10:47:24.412 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T10:47:24.414 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:47:24.414 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T10:47:24.421 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T10:47:24.433 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T10:47:24.437 INFO:teuthology.orchestra.run.smithi005.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T10:47:24.457 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T10:47:24.468 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T10:47:24.473 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T10:47:24.478 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T10:47:24.489 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T10:47:24.514 INFO:teuthology.orchestra.run.smithi005.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T10:47:24.531 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T10:47:24.534 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T10:47:24.555 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T10:47:24.567 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T10:47:24.636 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T10:47:25.024 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 28.7 kB in 1s (44.4 kB/s) 2024-09-16T10:47:25.097 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 28.7 kB in 1s (43.5 kB/s) 2024-09-16T10:47:26.176 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:47:26.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T10:47:26.270 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T10:47:26.289 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:47:26.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T10:47:26.391 INFO:teuthology.orchestra.run.smithi005.stdout:Reading package lists... 2024-09-16T10:47:26.489 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T10:47:26.489 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T10:47:26.591 INFO:teuthology.orchestra.run.smithi005.stdout:Building dependency tree... 2024-09-16T10:47:26.592 INFO:teuthology.orchestra.run.smithi005.stdout:Reading state information... 2024-09-16T10:47:26.691 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T10:47:26.693 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T10:47:26.694 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T10:47:26.694 INFO:teuthology.orchestra.run.smithi143.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T10:47:26.694 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T10:47:26.694 INFO:teuthology.orchestra.run.smithi143.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T10:47:26.694 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T10:47:26.695 INFO:teuthology.orchestra.run.smithi143.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T10:47:26.696 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T10:47:26.697 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-09-16T10:47:26.697 INFO:teuthology.orchestra.run.smithi143.stdout: btrfs-tools 2024-09-16T10:47:26.793 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T10:47:26.794 INFO:teuthology.orchestra.run.smithi005.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T10:47:26.794 INFO:teuthology.orchestra.run.smithi005.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T10:47:26.794 INFO:teuthology.orchestra.run.smithi005.stdout:The following additional packages will be installed: 2024-09-16T10:47:26.794 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T10:47:26.795 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T10:47:26.795 INFO:teuthology.orchestra.run.smithi005.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T10:47:26.795 INFO:teuthology.orchestra.run.smithi005.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T10:47:26.795 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T10:47:26.796 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T10:47:26.797 INFO:teuthology.orchestra.run.smithi005.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout:Suggested packages: 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T10:47:26.798 INFO:teuthology.orchestra.run.smithi005.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T10:47:26.799 INFO:teuthology.orchestra.run.smithi005.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T10:47:26.799 INFO:teuthology.orchestra.run.smithi005.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T10:47:26.799 INFO:teuthology.orchestra.run.smithi005.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T10:47:26.799 INFO:teuthology.orchestra.run.smithi005.stdout:Recommended packages: 2024-09-16T10:47:26.799 INFO:teuthology.orchestra.run.smithi005.stdout: btrfs-tools 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T10:47:26.822 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T10:47:26.823 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2024-09-16T10:47:26.824 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 librbd1 2024-09-16T10:47:26.920 INFO:teuthology.orchestra.run.smithi005.stdout:The following NEW packages will be installed: 2024-09-16T10:47:26.921 INFO:teuthology.orchestra.run.smithi005.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T10:47:26.921 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T10:47:26.921 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T10:47:26.923 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T10:47:26.924 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet 2024-09-16T10:47:26.925 INFO:teuthology.orchestra.run.smithi005.stdout:The following packages will be upgraded: 2024-09-16T10:47:26.925 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 librbd1 2024-09-16T10:47:26.970 INFO:teuthology.orchestra.run.smithi143.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T10:47:26.971 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 199 MB of archives. 2024-09-16T10:47:26.971 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T10:47:26.971 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T10:47:26.989 INFO:teuthology.orchestra.run.smithi005.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T10:47:26.989 INFO:teuthology.orchestra.run.smithi005.stdout:Need to get 199 MB of archives. 2024-09-16T10:47:26.990 INFO:teuthology.orchestra.run.smithi005.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T10:47:26.990 INFO:teuthology.orchestra.run.smithi005.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T10:47:27.015 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T10:47:27.074 INFO:teuthology.orchestra.run.smithi005.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T10:47:27.074 INFO:teuthology.orchestra.run.smithi005.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T10:47:27.076 INFO:teuthology.orchestra.run.smithi005.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T10:47:27.094 INFO:teuthology.orchestra.run.smithi005.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T10:47:27.148 INFO:teuthology.orchestra.run.smithi005.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T10:47:27.150 INFO:teuthology.orchestra.run.smithi005.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T10:47:27.156 INFO:teuthology.orchestra.run.smithi005.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T10:47:27.158 INFO:teuthology.orchestra.run.smithi005.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T10:47:27.158 INFO:teuthology.orchestra.run.smithi005.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T10:47:27.159 INFO:teuthology.orchestra.run.smithi005.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T10:47:27.160 INFO:teuthology.orchestra.run.smithi005.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T10:47:27.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T10:47:27.163 INFO:teuthology.orchestra.run.smithi005.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T10:47:27.165 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T10:47:27.165 INFO:teuthology.orchestra.run.smithi005.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T10:47:27.182 INFO:teuthology.orchestra.run.smithi005.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T10:47:27.183 INFO:teuthology.orchestra.run.smithi005.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T10:47:27.183 INFO:teuthology.orchestra.run.smithi005.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T10:47:27.185 INFO:teuthology.orchestra.run.smithi005.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T10:47:27.185 INFO:teuthology.orchestra.run.smithi005.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T10:47:27.185 INFO:teuthology.orchestra.run.smithi005.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T10:47:27.187 INFO:teuthology.orchestra.run.smithi005.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T10:47:27.187 INFO:teuthology.orchestra.run.smithi005.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T10:47:27.189 INFO:teuthology.orchestra.run.smithi005.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T10:47:27.201 INFO:teuthology.orchestra.run.smithi005.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T10:47:27.208 INFO:teuthology.orchestra.run.smithi005.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T10:47:27.215 INFO:teuthology.orchestra.run.smithi005.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T10:47:27.216 INFO:teuthology.orchestra.run.smithi005.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T10:47:27.217 INFO:teuthology.orchestra.run.smithi005.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T10:47:27.218 INFO:teuthology.orchestra.run.smithi005.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T10:47:27.220 INFO:teuthology.orchestra.run.smithi005.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T10:47:27.224 INFO:teuthology.orchestra.run.smithi005.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T10:47:27.224 INFO:teuthology.orchestra.run.smithi005.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T10:47:27.226 INFO:teuthology.orchestra.run.smithi005.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T10:47:27.232 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T10:47:27.235 INFO:teuthology.orchestra.run.smithi005.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T10:47:27.236 INFO:teuthology.orchestra.run.smithi005.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T10:47:27.237 INFO:teuthology.orchestra.run.smithi005.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T10:47:27.252 INFO:teuthology.orchestra.run.smithi005.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T10:47:27.252 INFO:teuthology.orchestra.run.smithi005.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T10:47:27.253 INFO:teuthology.orchestra.run.smithi005.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T10:47:27.255 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T10:47:27.256 INFO:teuthology.orchestra.run.smithi005.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T10:47:27.256 INFO:teuthology.orchestra.run.smithi005.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T10:47:27.256 INFO:teuthology.orchestra.run.smithi005.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T10:47:27.257 INFO:teuthology.orchestra.run.smithi005.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T10:47:27.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T10:47:27.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T10:47:27.265 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T10:47:27.265 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T10:47:27.269 INFO:teuthology.orchestra.run.smithi005.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T10:47:27.269 INFO:teuthology.orchestra.run.smithi005.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T10:47:27.271 INFO:teuthology.orchestra.run.smithi005.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T10:47:27.273 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T10:47:27.286 INFO:teuthology.orchestra.run.smithi005.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T10:47:27.288 INFO:teuthology.orchestra.run.smithi005.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T10:47:27.288 INFO:teuthology.orchestra.run.smithi005.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T10:47:27.289 INFO:teuthology.orchestra.run.smithi005.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T10:47:27.371 INFO:teuthology.orchestra.run.smithi005.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T10:47:27.388 INFO:teuthology.orchestra.run.smithi005.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T10:47:27.389 INFO:teuthology.orchestra.run.smithi005.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T10:47:27.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T10:47:27.401 INFO:teuthology.orchestra.run.smithi005.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T10:47:27.407 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T10:47:27.409 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T10:47:27.417 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T10:47:27.428 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T10:47:27.430 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T10:47:27.460 INFO:teuthology.orchestra.run.smithi005.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T10:47:27.461 INFO:teuthology.orchestra.run.smithi005.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T10:47:27.462 INFO:teuthology.orchestra.run.smithi005.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T10:47:27.464 INFO:teuthology.orchestra.run.smithi005.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T10:47:27.464 INFO:teuthology.orchestra.run.smithi005.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T10:47:27.466 INFO:teuthology.orchestra.run.smithi005.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T10:47:27.467 INFO:teuthology.orchestra.run.smithi005.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T10:47:27.478 INFO:teuthology.orchestra.run.smithi005.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T10:47:27.481 INFO:teuthology.orchestra.run.smithi005.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T10:47:27.483 INFO:teuthology.orchestra.run.smithi005.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T10:47:27.495 INFO:teuthology.orchestra.run.smithi005.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T10:47:27.496 INFO:teuthology.orchestra.run.smithi005.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T10:47:27.498 INFO:teuthology.orchestra.run.smithi005.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T10:47:27.501 INFO:teuthology.orchestra.run.smithi005.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T10:47:27.515 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T10:47:27.518 INFO:teuthology.orchestra.run.smithi005.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T10:47:27.519 INFO:teuthology.orchestra.run.smithi005.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T10:47:27.522 INFO:teuthology.orchestra.run.smithi005.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T10:47:27.524 INFO:teuthology.orchestra.run.smithi005.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T10:47:27.565 INFO:teuthology.orchestra.run.smithi005.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T10:47:27.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T10:47:27.566 INFO:teuthology.orchestra.run.smithi005.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T10:47:27.567 INFO:teuthology.orchestra.run.smithi005.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T10:47:27.578 INFO:teuthology.orchestra.run.smithi005.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T10:47:27.579 INFO:teuthology.orchestra.run.smithi005.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T10:47:27.580 INFO:teuthology.orchestra.run.smithi005.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T10:47:27.580 INFO:teuthology.orchestra.run.smithi005.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T10:47:27.580 INFO:teuthology.orchestra.run.smithi005.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T10:47:27.758 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T10:47:27.768 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T10:47:27.796 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T10:47:27.805 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T10:47:27.805 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T10:47:27.808 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T10:47:27.810 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T10:47:27.821 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T10:47:27.823 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T10:47:27.840 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T10:47:27.866 INFO:teuthology.orchestra.run.smithi005.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T10:47:27.879 INFO:teuthology.orchestra.run.smithi005.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T10:47:27.910 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T10:47:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T10:47:27.912 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T10:47:27.913 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T10:47:27.913 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T10:47:27.914 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T10:47:27.916 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T10:47:27.917 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T10:47:27.922 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T10:47:27.941 INFO:teuthology.orchestra.run.smithi005.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T10:47:27.942 INFO:teuthology.orchestra.run.smithi005.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T10:47:27.947 INFO:teuthology.orchestra.run.smithi005.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T10:47:27.949 INFO:teuthology.orchestra.run.smithi005.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T10:47:27.953 INFO:teuthology.orchestra.run.smithi005.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T10:47:28.003 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T10:47:28.054 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T10:47:28.055 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T10:47:28.060 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T10:47:28.061 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T10:47:28.122 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T10:47:28.123 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T10:47:28.124 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T10:47:28.128 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T10:47:28.129 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T10:47:28.134 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T10:47:28.167 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T10:47:28.170 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T10:47:28.243 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T10:47:28.246 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T10:47:28.248 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T10:47:28.248 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T10:47:28.249 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T10:47:28.250 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T10:47:28.251 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T10:47:28.298 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T10:47:28.298 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T10:47:28.298 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T10:47:28.299 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T10:47:28.300 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T10:47:28.330 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T10:47:28.330 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T10:47:28.373 INFO:teuthology.orchestra.run.smithi005.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T10:47:28.375 INFO:teuthology.orchestra.run.smithi005.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T10:47:28.410 INFO:teuthology.orchestra.run.smithi005.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T10:47:28.411 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T10:47:28.413 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T10:47:28.416 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T10:47:28.417 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T10:47:28.418 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T10:47:28.535 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T10:47:28.967 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T10:47:28.987 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T10:47:28.988 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T10:47:29.067 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T10:47:29.067 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T10:47:29.067 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T10:47:29.068 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T10:47:29.068 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T10:47:29.069 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T10:47:29.069 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T10:47:29.131 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T10:47:29.131 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T10:47:29.213 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T10:47:29.282 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T10:47:29.283 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T10:47:29.284 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T10:47:29.284 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T10:47:29.287 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T10:47:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T10:47:29.338 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T10:47:29.343 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T10:47:29.347 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T10:47:29.361 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T10:47:29.363 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T10:47:29.364 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T10:47:29.364 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T10:47:29.366 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T10:47:29.368 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T10:47:29.520 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T10:47:29.549 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T10:47:29.550 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T10:47:29.550 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T10:47:29.551 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T10:47:29.610 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T10:47:29.611 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T10:47:29.613 INFO:teuthology.orchestra.run.smithi143.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T10:47:29.613 INFO:teuthology.orchestra.run.smithi143.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T10:47:29.614 INFO:teuthology.orchestra.run.smithi143.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T10:47:29.640 INFO:teuthology.orchestra.run.smithi005.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T10:47:29.722 INFO:teuthology.orchestra.run.smithi143.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T10:47:29.722 INFO:teuthology.orchestra.run.smithi143.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T10:47:30.029 INFO:teuthology.orchestra.run.smithi005.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T10:47:30.032 INFO:teuthology.orchestra.run.smithi005.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T10:47:30.067 INFO:teuthology.orchestra.run.smithi005.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T10:47:30.132 INFO:teuthology.orchestra.run.smithi005.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T10:47:30.446 INFO:teuthology.orchestra.run.smithi005.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T10:47:31.202 INFO:teuthology.orchestra.run.smithi143.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T10:47:31.204 INFO:teuthology.orchestra.run.smithi143.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T10:47:31.205 INFO:teuthology.orchestra.run.smithi143.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T10:47:31.486 INFO:teuthology.orchestra.run.smithi143.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T10:47:31.548 INFO:teuthology.orchestra.run.smithi005.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T10:47:31.586 INFO:teuthology.orchestra.run.smithi005.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T10:47:31.601 INFO:teuthology.orchestra.run.smithi005.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T10:47:31.617 INFO:teuthology.orchestra.run.smithi005.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T10:47:31.619 INFO:teuthology.orchestra.run.smithi005.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T10:47:31.620 INFO:teuthology.orchestra.run.smithi005.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T10:47:31.729 INFO:teuthology.orchestra.run.smithi005.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T10:47:31.881 INFO:teuthology.orchestra.run.smithi005.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T10:47:31.881 INFO:teuthology.orchestra.run.smithi005.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T10:47:32.125 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 199 MB in 5s (42.7 MB/s) 2024-09-16T10:47:32.294 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T10:47:32.347 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T10:47:32.351 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T10:47:32.434 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T10:47:32.838 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T10:47:32.858 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T10:47:32.897 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T10:47:32.939 INFO:teuthology.orchestra.run.smithi005.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T10:47:32.942 INFO:teuthology.orchestra.run.smithi005.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T10:47:32.943 INFO:teuthology.orchestra.run.smithi005.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T10:47:33.118 INFO:teuthology.orchestra.run.smithi005.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T10:47:33.258 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T10:47:33.276 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T10:47:33.317 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T10:47:33.786 INFO:teuthology.orchestra.run.smithi005.stdout:Fetched 199 MB in 6s (32.2 MB/s) 2024-09-16T10:47:33.812 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T10:47:33.828 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:33.880 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:33.989 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T10:47:34.047 INFO:teuthology.orchestra.run.smithi005.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T10:47:34.051 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T10:47:34.138 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T10:47:34.423 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T10:47:34.441 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:34.482 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:34.483 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T10:47:34.501 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T10:47:34.533 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T10:47:34.860 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T10:47:34.876 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:34.877 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T10:47:34.897 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T10:47:34.918 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:34.935 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T10:47:35.313 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T10:47:35.329 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T10:47:35.397 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T10:47:35.423 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T10:47:35.439 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:35.491 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:35.757 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:35.884 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T10:47:35.984 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T10:47:36.000 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:36.034 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:36.420 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T10:47:36.438 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T10:47:36.479 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:47:36.612 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:36.738 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T10:47:36.882 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T10:47:36.901 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T10:47:36.940 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T10:47:37.287 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:37.385 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T10:47:37.467 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T10:47:37.487 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:37.525 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:37.895 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2024-09-16T10:47:37.911 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:37.953 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:38.115 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:38.241 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T10:47:38.323 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T10:47:38.341 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:38.381 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:38.609 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T10:47:38.628 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:38.667 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:38.962 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T10:47:38.980 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:39.012 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T10:47:39.020 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:39.028 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:39.070 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:39.331 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T10:47:39.351 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T10:47:39.390 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T10:47:39.423 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rados. 2024-09-16T10:47:39.441 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:39.473 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:39.659 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T10:47:39.677 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:39.717 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:39.800 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T10:47:39.820 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:39.883 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:40.087 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T10:47:40.105 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T10:47:40.136 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T10:47:40.145 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T10:47:40.152 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:40.194 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:40.423 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T10:47:40.440 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T10:47:40.481 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T10:47:40.514 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T10:47:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:40.564 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:40.843 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T10:47:40.863 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T10:47:40.892 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T10:47:40.901 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T10:47:40.909 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T10:47:40.950 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T10:47:41.228 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T10:47:41.245 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:41.271 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2024-09-16T10:47:41.286 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:41.290 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:41.346 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:41.664 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T10:47:41.684 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T10:47:41.722 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T10:47:42.050 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T10:47:42.068 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T10:47:42.133 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T10:47:42.157 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T10:47:42.176 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:42.206 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:42.461 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T10:47:42.478 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T10:47:42.519 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T10:47:42.576 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0. 2024-09-16T10:47:42.593 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T10:47:42.634 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T10:47:42.881 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package librgw2. 2024-09-16T10:47:42.901 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:42.939 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:42.988 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T10:47:43.005 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:43.046 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:43.699 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2024-09-16T10:47:43.716 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:43.758 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:43.825 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T10:47:43.842 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:43.908 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:44.219 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liboath0. 2024-09-16T10:47:44.235 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T10:47:44.278 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T10:47:44.622 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T10:47:44.639 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:44.680 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:45.401 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-common. 2024-09-16T10:47:45.420 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:45.458 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:46.289 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2024-09-16T10:47:46.305 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:46.374 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:47.627 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T10:47:47.643 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T10:47:47.685 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T10:47:47.997 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-base. 2024-09-16T10:47:48.014 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:48.039 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T10:47:48.056 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T10:47:48.069 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:48.113 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T10:47:48.425 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T10:47:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T10:47:48.483 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T10:47:48.820 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T10:47:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T10:47:48.878 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T10:47:49.182 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2024-09-16T10:47:49.197 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T10:47:49.272 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T10:47:49.368 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T10:47:49.385 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T10:47:49.427 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T10:47:49.593 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T10:47:49.609 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T10:47:49.651 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:47:49.788 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T10:47:49.806 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T10:47:49.847 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T10:47:49.979 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T10:47:49.996 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T10:47:50.037 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T10:47:50.275 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T10:47:50.296 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T10:47:50.315 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2024-09-16T10:47:50.335 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T10:47:50.391 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T10:47:50.508 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T10:47:50.721 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T10:47:50.736 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T10:47:50.776 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T10:47:50.811 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T10:47:50.829 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T10:47:50.869 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T10:47:51.113 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T10:47:51.129 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T10:47:51.155 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-mako. 2024-09-16T10:47:51.171 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T10:47:51.173 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T10:47:51.222 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T10:47:51.558 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T10:47:51.575 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T10:47:51.576 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T10:47:51.591 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T10:47:51.633 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:47:51.635 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T10:47:51.936 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T10:47:51.953 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T10:47:51.960 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T10:47:51.977 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T10:47:52.011 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T10:47:52.027 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T10:47:52.289 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webob. 2024-09-16T10:47:52.308 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T10:47:52.355 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T10:47:52.364 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2024-09-16T10:47:52.381 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T10:47:52.422 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T10:47:52.708 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T10:47:52.726 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T10:47:52.750 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T10:47:52.768 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T10:47:52.770 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T10:47:52.809 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T10:47:53.069 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T10:47:53.088 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T10:47:53.112 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T10:47:53.127 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T10:47:53.129 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T10:47:53.170 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T10:47:53.506 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T10:47:53.527 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T10:47:53.530 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T10:47:53.548 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T10:47:53.565 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T10:47:53.600 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T10:47:53.884 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T10:47:53.903 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T10:47:53.942 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T10:47:53.950 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T10:47:53.969 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T10:47:54.009 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T10:47:54.264 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T10:47:54.283 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T10:47:54.323 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T10:47:54.328 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-paste. 2024-09-16T10:47:54.346 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T10:47:54.386 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T10:47:54.634 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T10:47:54.653 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:54.682 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T10:47:54.692 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:54.698 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T10:47:54.765 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T10:47:55.067 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T10:47:55.072 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T10:47:55.086 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T10:47:55.092 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:55.126 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T10:47:55.130 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:55.458 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T10:47:55.462 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T10:47:55.475 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T10:47:55.478 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T10:47:55.517 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T10:47:55.545 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T10:47:55.845 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T10:47:55.861 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T10:47:55.873 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T10:47:55.893 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T10:47:55.903 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T10:47:55.940 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T10:47:56.307 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T10:47:56.318 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T10:47:56.325 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T10:47:56.335 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T10:47:56.365 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T10:47:56.376 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T10:47:56.688 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T10:47:56.702 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T10:47:56.703 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:47:56.718 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:56.746 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:56.769 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:57.143 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T10:47:57.159 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:57.201 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:57.256 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T10:47:57.273 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:57.330 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:57.521 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T10:47:57.538 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T10:47:57.578 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T10:47:57.940 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T10:47:57.960 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T10:47:58.114 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T10:47:58.264 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T10:47:58.279 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:58.321 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:58.512 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T10:47:58.527 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T10:47:58.569 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T10:47:58.938 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T10:47:58.957 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:58.998 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:47:59.529 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T10:47:59.546 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:47:59.587 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:00.466 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T10:48:00.485 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:00.531 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:00.775 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2024-09-16T10:48:00.792 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:01.177 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T10:48:01.195 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:01.235 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:01.681 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T10:48:01.699 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:01.738 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:02.291 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2024-09-16T10:48:02.311 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:02.349 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:02.677 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T10:48:02.696 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T10:48:02.735 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T10:48:02.993 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph. 2024-09-16T10:48:03.012 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:03.051 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:03.090 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T10:48:03.108 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T10:48:03.148 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T10:48:03.320 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T10:48:03.338 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:03.378 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:03.476 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T10:48:03.493 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:03.534 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:03.773 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T10:48:03.793 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:03.831 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:03.896 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T10:48:03.912 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T10:48:03.956 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T10:48:04.226 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-attr. 2024-09-16T10:48:04.242 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T10:48:04.284 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T10:48:04.377 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package cephadm. 2024-09-16T10:48:04.394 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:04.437 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:04.637 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T10:48:04.653 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T10:48:04.728 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T10:48:04.779 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T10:48:04.800 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T10:48:04.837 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T10:48:05.115 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2024-09-16T10:48:05.135 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T10:48:05.173 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T10:48:05.249 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T10:48:05.268 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T10:48:05.307 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T10:48:05.485 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T10:48:05.504 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T10:48:05.568 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T10:48:05.660 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T10:48:05.680 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:05.727 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:05.871 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T10:48:05.889 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T10:48:05.929 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T10:48:06.063 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T10:48:06.079 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T10:48:06.138 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T10:48:06.265 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T10:48:06.284 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T10:48:06.323 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T10:48:06.433 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-attr. 2024-09-16T10:48:06.452 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T10:48:06.491 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T10:48:06.627 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T10:48:06.645 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T10:48:06.718 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T10:48:06.902 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T10:48:06.919 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T10:48:06.960 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T10:48:07.046 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T10:48:07.063 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T10:48:07.105 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T10:48:07.272 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-py. 2024-09-16T10:48:07.288 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T10:48:07.322 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T10:48:07.391 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T10:48:07.409 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T10:48:07.449 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T10:48:07.633 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T10:48:07.650 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T10:48:07.683 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T10:48:07.728 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T10:48:07.745 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T10:48:07.778 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T10:48:08.019 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T10:48:08.036 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T10:48:08.077 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T10:48:08.081 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2024-09-16T10:48:08.097 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T10:48:08.131 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T10:48:08.405 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T10:48:08.422 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T10:48:08.463 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T10:48:08.493 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T10:48:08.509 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:08.551 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:08.775 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T10:48:08.791 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T10:48:08.850 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T10:48:09.171 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T10:48:09.189 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T10:48:09.221 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T10:48:09.549 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T10:48:09.566 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T10:48:09.607 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T10:48:09.909 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T10:48:09.926 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T10:48:09.948 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T10:48:09.965 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T10:48:09.967 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T10:48:10.015 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T10:48:10.312 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-routes. 2024-09-16T10:48:10.329 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T10:48:10.343 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T10:48:10.359 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T10:48:10.370 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T10:48:10.401 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T10:48:10.723 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T10:48:10.740 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:10.758 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T10:48:10.776 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T10:48:10.781 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:10.820 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T10:48:12.105 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T10:48:12.121 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T10:48:12.163 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:12.253 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T10:48:12.271 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T10:48:12.311 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T10:48:12.648 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T10:48:12.666 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T10:48:12.706 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T10:48:12.725 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T10:48:12.741 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T10:48:12.783 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T10:48:13.059 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T10:48:13.075 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T10:48:13.117 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T10:48:13.195 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T10:48:13.212 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T10:48:13.253 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:13.962 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T10:48:13.980 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:14.020 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:14.468 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T10:48:14.484 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T10:48:14.525 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:15.079 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T10:48:15.101 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T10:48:15.154 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T10:48:15.508 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T10:48:15.525 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T10:48:15.557 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T10:48:15.565 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T10:48:15.574 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T10:48:15.615 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:15.877 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T10:48:15.893 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T10:48:15.935 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T10:48:16.305 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T10:48:16.321 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T10:48:16.336 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T10:48:16.356 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:16.363 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T10:48:16.393 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:16.770 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T10:48:16.788 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T10:48:16.828 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T10:48:17.132 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T10:48:17.149 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T10:48:17.215 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T10:48:17.513 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T10:48:17.523 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T10:48:17.533 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T10:48:17.543 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T10:48:17.571 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T10:48:17.581 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T10:48:17.908 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T10:48:17.918 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T10:48:17.924 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T10:48:17.938 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T10:48:17.966 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T10:48:17.976 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T10:48:18.312 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T10:48:18.326 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T10:48:18.328 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T10:48:18.344 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T10:48:18.370 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T10:48:18.388 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T10:48:18.683 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T10:48:18.703 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T10:48:18.741 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T10:48:18.850 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T10:48:19.070 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T10:48:19.088 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T10:48:19.128 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T10:48:19.191 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T10:48:19.207 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T10:48:19.249 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T10:48:19.562 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T10:48:19.581 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T10:48:19.620 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T10:48:19.693 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T10:48:19.709 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:19.751 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:19.948 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T10:48:19.964 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T10:48:20.006 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T10:48:20.117 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T10:48:20.133 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T10:48:20.175 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T10:48:20.285 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T10:48:20.304 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T10:48:20.334 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T10:48:20.528 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T10:48:20.545 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T10:48:20.586 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:20.655 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T10:48:20.671 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T10:48:20.713 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T10:48:20.873 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2024-09-16T10:48:20.892 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T10:48:20.931 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:21.058 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T10:48:21.075 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T10:48:21.116 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T10:48:21.185 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2024-09-16T10:48:21.201 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T10:48:21.259 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T10:48:21.565 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T10:48:21.584 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T10:48:21.623 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T10:48:21.650 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T10:48:21.667 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:21.708 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:21.995 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2024-09-16T10:48:22.011 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:22.053 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:22.265 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T10:48:22.279 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T10:48:22.320 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T10:48:22.690 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T10:48:22.706 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T10:48:22.748 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:23.028 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package jq. 2024-09-16T10:48:23.044 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T10:48:23.119 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:23.372 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package socat. 2024-09-16T10:48:23.390 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T10:48:23.422 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T10:48:23.751 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T10:48:23.769 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T10:48:23.808 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T10:48:24.163 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-test. 2024-09-16T10:48:24.181 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:24.213 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:26.116 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T10:48:26.135 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:26.173 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:26.493 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T10:48:26.512 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T10:48:26.552 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:27.081 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T10:48:27.100 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:27.139 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:27.534 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T10:48:27.555 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T10:48:27.602 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T10:48:27.931 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T10:48:27.949 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T10:48:27.989 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T10:48:28.350 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T10:48:28.360 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T10:48:28.368 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T10:48:28.377 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T10:48:28.408 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:28.419 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:28.721 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T10:48:28.737 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T10:48:28.778 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:29.399 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T10:48:29.415 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:29.448 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:29.844 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T10:48:29.864 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T10:48:29.920 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T10:48:30.061 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T10:48:30.081 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T10:48:30.126 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T10:48:30.251 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T10:48:30.271 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T10:48:30.309 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T10:48:30.462 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T10:48:30.481 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T10:48:30.553 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T10:48:30.696 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T10:48:30.714 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T10:48:30.754 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:30.839 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T10:48:30.860 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T10:48:30.898 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T10:48:31.259 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T10:48:31.277 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T10:48:31.317 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T10:48:31.828 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T10:48:31.846 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T10:48:31.886 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T10:48:32.214 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T10:48:32.231 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T10:48:32.281 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T10:48:32.430 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T10:48:32.451 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T10:48:32.487 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T10:48:32.609 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T10:48:32.626 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T10:48:32.667 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T10:48:32.849 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T10:48:32.866 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T10:48:32.907 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T10:48:33.003 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T10:48:33.019 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T10:48:33.061 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T10:48:33.235 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T10:48:33.254 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T10:48:33.293 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T10:48:33.355 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T10:48:33.431 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T10:48:33.438 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T10:48:33.654 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T10:48:33.673 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T10:48:33.712 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T10:48:34.094 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2024-09-16T10:48:34.114 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:34.152 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:34.183 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T10:48:34.199 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T10:48:34.240 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T10:48:34.603 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T10:48:34.619 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T10:48:34.660 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T10:48:35.031 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T10:48:35.050 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T10:48:35.088 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T10:48:35.413 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T10:48:35.433 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T10:48:35.477 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T10:48:35.547 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T10:48:35.564 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:35.605 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:35.738 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T10:48:35.755 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T10:48:35.796 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T10:48:35.956 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T10:48:36.251 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T10:48:36.402 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T10:48:36.509 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package radosgw. 2024-09-16T10:48:36.527 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:36.528 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T10:48:36.567 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:36.729 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:37.241 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2024-09-16T10:48:37.404 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:37.555 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T10:48:37.915 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T10:48:38.003 INFO:teuthology.orchestra.run.smithi005.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T10:48:38.020 INFO:teuthology.orchestra.run.smithi005.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T10:48:38.061 INFO:teuthology.orchestra.run.smithi005.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:38.241 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:38.366 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T10:48:38.388 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T10:48:38.492 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T10:48:38.708 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T10:48:38.819 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T10:48:38.854 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T10:48:38.945 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T10:48:39.180 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:39.280 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T10:48:39.582 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T10:48:39.731 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user cephadm....done 2024-09-16T10:48:39.883 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T10:48:39.895 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:40.054 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T10:48:40.167 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T10:48:40.415 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T10:48:40.451 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:40.744 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:40.750 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T10:48:40.851 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T10:48:40.920 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T10:48:40.985 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T10:48:41.204 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T10:48:41.328 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T10:48:41.497 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T10:48:41.623 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T10:48:41.789 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T10:48:41.999 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T10:48:42.124 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T10:48:42.316 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T10:48:42.425 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T10:48:42.626 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T10:48:42.684 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T10:48:42.960 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T10:48:42.994 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:43.228 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T10:48:43.303 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:43.429 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T10:48:43.571 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T10:48:43.714 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T10:48:43.872 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T10:48:44.015 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T10:48:44.166 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T10:48:44.173 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T10:48:44.299 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T10:48:44.449 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T10:48:44.559 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T10:48:44.734 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T10:48:44.843 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T10:48:45.110 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T10:48:45.128 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T10:48:45.492 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T10:48:45.496 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T10:48:45.789 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T10:48:45.855 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T10:48:46.157 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T10:48:46.450 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T10:48:46.528 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:46.751 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T10:48:46.893 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T10:48:46.907 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:47.003 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T10:48:47.041 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T10:48:47.304 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T10:48:47.393 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T10:48:47.518 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T10:48:47.644 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T10:48:47.811 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T10:48:47.946 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T10:48:48.272 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:48.301 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T10:48:48.398 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T10:48:48.548 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T10:48:48.653 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T10:48:48.779 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:48:48.875 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T10:48:49.122 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:49.151 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T10:48:49.256 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T10:48:49.452 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T10:48:49.578 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:48:49.608 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T10:48:49.726 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T10:48:49.826 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T10:48:49.887 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T10:48:50.119 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T10:48:50.171 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T10:48:50.438 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:50.472 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T10:48:50.563 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T10:48:50.689 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T10:48:50.841 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T10:48:51.058 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T10:48:51.116 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T10:48:51.242 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T10:48:51.342 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T10:48:51.543 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T10:48:51.618 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T10:48:51.745 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:48:51.870 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T10:48:52.021 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T10:48:52.037 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T10:48:52.147 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T10:48:52.338 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T10:48:52.473 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T10:48:52.673 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T10:48:52.782 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T10:48:53.016 INFO:teuthology.orchestra.run.smithi005.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T10:48:53.476 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T10:48:53.602 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T10:48:53.911 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T10:48:54.279 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:54.296 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T10:48:54.405 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T10:48:54.464 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T10:48:54.590 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T10:48:54.773 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T10:48:54.924 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T10:48:55.108 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T10:48:55.234 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T10:48:55.425 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:55.526 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T10:48:55.827 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T10:48:56.146 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T10:48:56.514 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T10:48:56.773 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:56.865 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:56.899 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T10:48:57.284 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T10:48:57.594 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T10:48:57.886 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T10:48:57.887 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T10:48:58.013 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T10:48:58.030 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T10:48:58.139 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:48:58.264 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T10:48:58.380 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T10:48:58.591 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T10:48:58.717 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T10:48:58.741 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T10:48:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T10:48:59.133 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T10:48:59.310 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T10:48:59.443 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T10:48:59.620 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T10:48:59.964 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:00.123 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T10:49:00.249 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:00.375 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:00.540 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T10:49:00.595 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:00.658 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T10:49:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T10:49:00.809 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T10:49:00.935 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T10:49:01.030 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:01.156 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T10:49:01.237 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T10:49:01.362 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T10:49:01.483 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:01.608 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:01.663 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T10:49:01.734 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:01.972 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T10:49:01.991 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:49:01.991 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:49:02.324 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T10:49:02.427 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:02.562 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:02.687 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T10:49:02.810 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T10:49:02.936 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.087 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.299 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.475 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T10:49:03.616 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.742 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.776 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.868 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:03.927 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T10:49:03.994 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:04.162 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:04.237 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:04.380 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:04.506 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:04.731 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2024-09-16T10:49:04.732 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:49:04.732 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:49:05.107 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2024-09-16T10:49:05.150 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:05.300 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2024-09-16T10:49:05.300 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:05.325 INFO:teuthology.orchestra.run.smithi143.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T10:49:05.426 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T10:49:05.472 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T10:49:05.760 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T10:49:06.246 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.347 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.364 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.490 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.649 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.775 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:06.824 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:49:06.824 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:49:06.901 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:07.257 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:07.579 INFO:teuthology.orchestra.run.smithi005.stdout:Adding group ceph....done 2024-09-16T10:49:07.635 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T10:49:07.992 INFO:teuthology.orchestra.run.smithi005.stdout:Adding system user ceph....done 2024-09-16T10:49:08.058 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:08.157 INFO:teuthology.orchestra.run.smithi005.stdout:Setting system user ceph properties....done 2024-09-16T10:49:08.181 INFO:teuthology.orchestra.run.smithi005.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T10:49:08.313 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:49:08.313 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:49:08.343 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T10:49:08.676 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T10:49:08.760 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:08.994 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:49:08.994 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:49:09.445 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:09.570 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:09.697 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:09.945 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:49:09.945 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:49:10.189 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:49:10.189 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:49:10.438 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:10.564 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:10.644 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:10.706 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:10.926 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:49:10.926 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:49:11.048 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T10:49:11.357 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:11.492 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:11.500 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:11.652 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:11.775 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:49:11.775 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:49:11.793 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:12.188 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T10:49:12.252 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:12.483 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:49:12.483 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:49:12.566 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T10:49:12.859 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T10:49:12.919 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:13.424 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:49:13.425 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:49:13.863 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:13.989 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:14.124 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:14.353 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:49:14.353 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:49:14.800 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:14.943 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:15.069 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:15.228 INFO:teuthology.orchestra.run.smithi005.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T10:49:15.573 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T10:49:15.686 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T10:49:15.750 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T10:49:16.034 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T10:49:16.294 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T10:49:16.300 DEBUG:teuthology.parallel:result is None 2024-09-16T10:49:18.954 INFO:teuthology.orchestra.run.smithi005.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T10:49:19.563 INFO:teuthology.orchestra.run.smithi005.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T10:49:19.568 DEBUG:teuthology.parallel:result is None 2024-09-16T10:49:19.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:49:19.714 DEBUG:teuthology.orchestra.run.smithi005:> dpkg-query -W -f '${Version}' ceph 2024-09-16T10:49:19.739 INFO:teuthology.orchestra.run.smithi005.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T10:49:19.740 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T10:49:19.740 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T10:49:19.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:49:19.877 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2024-09-16T10:49:19.902 INFO:teuthology.orchestra.run.smithi143.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T10:49:19.903 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T10:49:19.903 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T10:49:19.905 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T10:49:19.905 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:19.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:49:19.922 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:19.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:49:19.961 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T10:49:19.961 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:19.961 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:49:19.983 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:49:19.993 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:19.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:49:20.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:49:20.019 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T10:49:20.020 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:20.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:49:20.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:49:20.090 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:20.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:49:20.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:49:20.122 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T10:49:20.122 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:20.122 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:49:20.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:49:20.149 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:20.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:49:20.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:49:20.192 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T10:49:20.289 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T10:49:20.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-09-16T10:49:20.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2024-09-16T10:49:20.308 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T10:49:20.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T10:49:20.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T10:49:20.365 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T10:49:20.366 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T10:49:20.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T10:49:20.396 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T10:49:20.397 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T10:49:20.397 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906688', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T10:49:20.397 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T10:49:20.400 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T10:49:20.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T10:49:20.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T10:49:20.466 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:20.466 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:49:20.471 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:49:20.471 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-09-16T10:49:20.477 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:49:20.477 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-16T10:49:20.477 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-16 10:45:49.374606687 +0000 2024-09-16T10:49:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-16 10:45:49.162611567 +0000 2024-09-16T10:49:20.478 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-16 10:45:49.162611567 +0000 2024-09-16T10:49:20.478 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-16T10:49:20.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:49:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-16T10:49:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-16T10:49:20.533 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000312608 s, 1.6 MB/s 2024-09-16T10:49:20.534 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:49:20.543 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-16 10:45:49.786597204 +0000 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-16 10:45:49.570602176 +0000 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-16 10:45:49.570602176 +0000 2024-09-16T10:49:20.551 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-16T10:49:20.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:49:20.613 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-16T10:49:20.613 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-16T10:49:20.613 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00029833 s, 1.7 MB/s 2024-09-16T10:49:20.614 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:49:20.622 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-09-16T10:49:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:49:20.626 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.626 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-16T10:49:20.626 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.627 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-16 10:45:50.250586523 +0000 2024-09-16T10:49:20.627 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-16 10:45:50.010592047 +0000 2024-09-16T10:49:20.627 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-16 10:45:50.010592047 +0000 2024-09-16T10:49:20.627 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-16T10:49:20.627 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:49:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-16T10:49:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-16T10:49:20.684 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000294441 s, 1.7 MB/s 2024-09-16T10:49:20.686 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:49:20.693 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-16 10:45:50.666576948 +0000 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-16 10:45:50.434582288 +0000 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-16 10:45:50.434582288 +0000 2024-09-16T10:49:20.702 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: - 2024-09-16T10:49:20.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:49:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-16T10:49:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-16T10:49:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000295791 s, 1.7 MB/s 2024-09-16T10:49:20.759 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:49:20.766 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:20.766 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:20.766 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:49:20.772 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:49:20.773 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 369 Links: 1 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 10:45:49.136782370 +0000 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 10:45:49.132782462 +0000 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 10:45:49.132782462 +0000 2024-09-16T10:49:20.780 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T10:49:20.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:49:20.833 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T10:49:20.833 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T10:49:20.834 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00041084 s, 1.2 MB/s 2024-09-16T10:49:20.834 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:49:20.843 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-09-16T10:49:20.849 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:49:20.849 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.849 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-16T10:49:20.850 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.850 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 10:45:49.848765958 +0000 2024-09-16T10:49:20.850 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 10:45:49.612771399 +0000 2024-09-16T10:49:20.850 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 10:45:49.612771399 +0000 2024-09-16T10:49:20.850 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T10:49:20.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:49:20.863 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T10:49:20.863 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T10:49:20.863 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000413685 s, 1.2 MB/s 2024-09-16T10:49:20.865 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:49:20.918 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 384 Links: 1 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 10:45:50.240756923 +0000 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 10:45:50.040761533 +0000 2024-09-16T10:49:20.925 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 10:45:50.040761533 +0000 2024-09-16T10:49:20.926 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T10:49:20.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:49:20.938 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T10:49:20.938 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T10:49:20.938 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000397925 s, 1.3 MB/s 2024-09-16T10:49:20.939 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:49:20.947 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 389 Links: 1 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 10:45:50.700746320 +0000 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 10:45:50.484751298 +0000 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 10:45:50.484751298 +0000 2024-09-16T10:49:20.996 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T10:49:20.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:49:21.009 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T10:49:21.009 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T10:49:21.009 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000384078 s, 1.3 MB/s 2024-09-16T10:49:21.010 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:49:21.019 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:21.019 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:49:21.019 INFO:tasks.ceph:Generating config... 2024-09-16T10:49:21.020 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T10:49:21.021 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] debug ms = 1 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] debug rados = 20 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T10:49:21.021 INFO:tasks.ceph:[global] ms type = async 2024-09-16T10:49:21.021 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T10:49:21.021 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T10:49:21.021 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T10:49:21.022 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T10:49:21.022 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T10:49:21.022 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T10:49:21.023 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T10:49:21.024 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T10:49:21.024 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T10:49:21.025 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T10:49:21.067 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T10:49:21.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T10:49:21.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:49:21.184 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5'), ('mon.c', '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]'), ('mon.b', '172.21.15.143')] 2024-09-16T10:49:21.185 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.143', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T10:49:21.186 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:21.186 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:49:21.234 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --addv c '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]' --add b 172.21.15.143 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:21.280 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = pacific 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 16 (pacific) 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 3 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:21.281 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T10:49:21.283 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:49:21.288 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0ea84356-fa95-422b-b956-38c4a6b7c43e... 2024-09-16T10:49:21.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T10:49:21.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T10:49:21.365 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-09-16T10:49:21.366 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-09-16T10:49:21.367 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.368 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.143" 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: debug objecter = 20 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: debug rados = 20 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: mon election default strategy = 3 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr1 = False 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr2 = True 2024-09-16T10:49:21.369 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 5000 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: ms type = async 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.370 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.371 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-09-16T10:49:21.372 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore zero block detection = True 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd class default list = * 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd class load list = * 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify cached snaps = True 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify missing on start = True 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-09-16T10:49:21.373 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock override recovery settings = True 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue = debug_random 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue cut off = debug_random 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.374 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T10:49:21.375 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon min osdmap epochs = 25 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune interval = 2 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune min = 15 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune txsize = 2 2024-09-16T10:49:21.376 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub inject missing keys = 0.05 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub interval = 300 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on pool no app = False 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: paxos service trim min = 5 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-09-16T10:49:21.377 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-16T10:49:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-09-16T10:49:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-09-16T10:49:21.378 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-09-16T10:49:21.379 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-09-16T10:49:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2024-09-16T10:49:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T10:49:21.379 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.380 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2024-09-16T10:49:21.381 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2024-09-16T10:49:21.382 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.5,[v2:172.21.15.5:3301,v1:172.21.15.5:6790],172.21.15.143" 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: debug objecter = 20 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: debug rados = 20 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: mon election default strategy = 3 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: ms bind msgr1 = False 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: ms bind msgr2 = True 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: ms type = async 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:21.383 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2024-09-16T10:49:21.384 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.385 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bdev async discard = True 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bdev enable discard = True 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore allocator = bitmap 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore zero block detection = True 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T10:49:21.386 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd class default list = * 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd class load list = * 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify cached snaps = True 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify missing on start = True 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock override recovery settings = True 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue = debug_random 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue cut off = debug_random 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T10:49:21.387 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-16T10:49:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T10:49:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon min osdmap epochs = 25 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune interval = 2 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune min = 15 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune txsize = 2 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject missing keys = 0.05 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub interval = 300 2024-09-16T10:49:21.390 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on pool no app = False 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: paxos service trim min = 5 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: debug monc = 20 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2024-09-16T10:49:21.391 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2024-09-16T10:49:21.392 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T10:49:21.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T10:49:21.447 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T10:49:21.448 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:21.448 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T10:49:21.453 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:21.453 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T10:49:21.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T10:49:21.459 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:21.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T10:49:21.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:49:21.482 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:21.482 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:21.489 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T10:49:21.489 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:21.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T10:49:21.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:49:21.518 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:21.518 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:21.522 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T10:49:21.522 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T10:49:21.522 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:49:21.615 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:49:21.626 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T10:49:21.627 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T10:49:21.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T10:49:21.666 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T10:49:21.678 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T10:49:21.678 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:49:21.678 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T10:49:21.695 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:21.695 INFO:tasks.ceph:role: osd.0 2024-09-16T10:49:21.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T10:49:21.695 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T10:49:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:21.782 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:21.782 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:21.782 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-16T10:49:21.782 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:21.783 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:21.783 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:21.783 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:21.784 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-16T10:49:21.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T10:49:21.804 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T10:49:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:21.815 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:21.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T10:49:21.874 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:21.874 INFO:tasks.ceph:role: osd.1 2024-09-16T10:49:21.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T10:49:21.874 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T10:49:21.917 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:21.917 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:21.917 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:21.918 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:21.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-16T10:49:21.919 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T10:49:21.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T10:49:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:21.948 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:21.949 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T10:49:22.007 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:22.007 INFO:tasks.ceph:role: osd.2 2024-09-16T10:49:22.007 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T10:49:22.007 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T10:49:22.050 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:22.051 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:22.052 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-16T10:49:22.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T10:49:22.115 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T10:49:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:22.126 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:22.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:22.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T10:49:22.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T10:49:22.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:49:22.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:49:22.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:49:22.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:22.189+0000 7fb122a24880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T10:49:25.183 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T10:49:25.199 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:25.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.281+0000 7fc08540f880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T10:49:25.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.281+0000 7fc08540f880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T10:49:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.281+0000 7fc08540f880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:49:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.281+0000 7fc08540f880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:49:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.281+0000 7fc08540f880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:49:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:25.285+0000 7fc08540f880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T10:49:28.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T10:49:28.299 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T10:49:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T10:49:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:49:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:49:28.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:49:28.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:28.333+0000 7f30c5d28880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T10:49:31.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T10:49:31.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T10:49:31.360 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:31.360 INFO:tasks.ceph:role: osd.3 2024-09-16T10:49:31.360 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T10:49:31.360 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T10:49:31.903 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:31.903 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:31.904 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:31.905 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T10:49:31.906 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T10:49:32.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T10:49:32.017 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:32.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T10:49:32.034 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:32.034 INFO:tasks.ceph:role: osd.4 2024-09-16T10:49:32.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T10:49:32.034 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T10:49:32.561 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:32.562 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:32.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T10:49:32.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T10:49:32.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T10:49:32.654 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:32.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T10:49:32.667 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T10:49:32.667 INFO:tasks.ceph:role: osd.5 2024-09-16T10:49:32.667 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T10:49:32.667 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:49:33.125 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:49:33.126 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T10:49:33.126 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:49:33.126 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:49:33.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T10:49:33.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T10:49:33.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T10:49:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:49:33.193 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T10:49:33.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:33.277 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.274+0000 7f7014061880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T10:49:33.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.274+0000 7f7014061880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T10:49:33.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.274+0000 7f7014061880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:49:33.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.274+0000 7f7014061880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:49:33.278 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.274+0000 7f7014061880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T10:49:33.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:33.278+0000 7f7014061880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T10:49:36.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T10:49:36.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:36.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.342+0000 7f55b3674880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T10:49:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.342+0000 7f55b3674880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T10:49:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.346+0000 7f55b3674880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:49:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.346+0000 7f55b3674880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:49:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.346+0000 7f55b3674880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T10:49:36.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:36.346+0000 7f55b3674880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T10:49:39.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T10:49:39.401 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:39.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T10:49:39.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T10:49:39.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:49:39.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:49:39.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T10:49:39.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:39.438+0000 7f1f36eb6880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T10:49:42.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T10:49:42.458 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T10:49:42.458 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:42.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T10:49:42.474 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:42.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T10:49:42.485 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:42.486 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T10:49:42.496 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:42.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T10:49:42.516 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:42.516 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T10:49:42.528 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:42.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T10:49:42.539 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:42.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T10:49:42.596 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:42.596 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T10:49:42.603 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T10:49:42.603 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T10:49:42.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T10:49:42.617 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAyDehmmnKnCxAAqdEyqBS/IR4wSoe+WGEMpA== 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA1Dehm0tEtERAASSISvVUjf1NjJVsaGYs+kA== 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA4DehmPL4/FBAAzKPJranu2QjiBixhG1IyUg== 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAxDehmIKKtJBAA2pno9EKbeI9htEYNf3sJwg== 2024-09-16T10:49:42.618 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA9Dehmw1SXEBAAivtrciElnMUPSLjcAEeTrg== 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBADehmY2DVFBAA/jOx9k6Xsb2jFpZMNK9ZSQ== 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBDDehmSGhoGhAAFMRxY8mZeu/ew1Ue6YMSEw== 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2024-09-16T10:49:42.619 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAxDehmf9W+JxAAB3zhCkjun6QFJGIyikO5mw== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAyDehmmnKnCxAAqdEyqBS/IR4wSoe+WGEMpA== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA1Dehm0tEtERAASSISvVUjf1NjJVsaGYs+kA== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA4DehmPL4/FBAAzKPJranu2QjiBixhG1IyUg== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAxDehmIKKtJBAA2pno9EKbeI9htEYNf3sJwg== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQA9Dehmw1SXEBAAivtrciElnMUPSLjcAEeTrg== 2024-09-16T10:49:42.621 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-09-16T10:49:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBADehmY2DVFBAA/jOx9k6Xsb2jFpZMNK9ZSQ== 2024-09-16T10:49:42.622 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-09-16T10:49:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBDDehmSGhoGhAAFMRxY8mZeu/ew1Ue6YMSEw== 2024-09-16T10:49:42.622 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-09-16T10:49:42.622 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQAxDehmf9W+JxAAB3zhCkjun6QFJGIyikO5mw== 2024-09-16T10:49:42.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T10:49:42.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T10:49:42.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.921 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.972 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:42.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:49:43.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T10:49:43.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T10:49:43.076 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T10:49:43.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T10:49:43.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:49:43.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T10:49:43.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T10:49:43.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:49:43.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T10:49:43.302 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T10:49:43.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:49:43.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T10:49:43.470 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:43.473 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:49:43.477 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T10:49:43.478 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T10:49:43.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T10:49:43.480 INFO:tasks.ceph.mon.a:Started 2024-09-16T10:49:43.480 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T10:49:43.480 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T10:49:43.482 INFO:tasks.ceph.mon.c:Started 2024-09-16T10:49:43.483 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:49:43.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:49:43.485 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:49:43.485 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T10:49:43.485 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T10:49:43.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T10:49:43.488 INFO:tasks.ceph.mgr.x:Started 2024-09-16T10:49:43.488 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T10:49:43.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T10:49:43.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: get_monmap_and_config 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: build_initial_monmap 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: monmap: 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:43.595085+0000 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:43.595085+0000 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:43.598 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 Processor -- start 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 -- start start 2024-09-16T10:49:43.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: init 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bcc09baa0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-b con 0x7f2bcc09be80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bcc0a14e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0990b0 con 0x7f2bcc0a14e0 2024-09-16T10:49:43.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc134470 con 0x7f2bcc09baa0 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc073c20 con 0x7f2bcc09be80 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd29e8700 10 monclient(hunting): authenticate will time out at 842.171534s 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:43.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd0f85700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2bcc0a14e0 0x7f2bcc13bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35700/0 (socket says 172.21.15.5:35700) 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 learned_addr learned my addr 172.21.15.5:0/2940672002 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38518/0 (socket says 172.21.15.5:38518) 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc09be80 auth_method 0 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:43.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.589+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:43.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:43.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc09be80 auth_method 0 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:43.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:43.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:49:44.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc09be80 auth_method 0 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:44.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:44.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc09be80 auth_method 0 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:45.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:45.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:45.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:49:46.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): tick 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 monclient(hunting): continuing hunt 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 msgr2=0x7f2bcc133f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc09be80 0x7f2bcc133f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 >> v1:172.21.15.143:6789/0 conn(0x7f2bcc09baa0 legacy=0x7f2bcc0a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 >> v1:172.21.15.5:6789/0 conn(0x7f2bcc0a14e0 legacy=0x7f2bcc13bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient: _add_conns ranks=[3,4] 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f2bb8003790 addr v2:172.21.15.5:3300/0 2024-09-16T10:49:46.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f2bb80062e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc073c20 con 0x7f2bb8003790 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb8008e60 con 0x7f2bb80062e0 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.589+0000 7f2bc9ffb700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb8003790 auth_method 0 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:46.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:46.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.593+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb8003790 auth_method 0 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:46.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:46.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:46.797+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb8003790 auth_method 0 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:47.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:47.201+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:49:48.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:48.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb8003790 auth_method 0 2024-09-16T10:49:48.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:48.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:48.005+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:49:49.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): tick 2024-09-16T10:49:49.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 monclient(hunting): continuing hunt 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 msgr2=0x7f2bb8005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.5:3300/0 conn(0x7f2bb8003790 0x7f2bb8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 reap_dead start 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f2bb80062e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:49.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bcc0a14e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f2bb8003be0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc09f490 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc073c20 con 0x7f2bb80062e0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 1 -- 172.21.15.5:0/2940672002 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8008e60 con 0x7f2bb8003be0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bc9ffb700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request con 0x7f2bb8003be0 auth_method 0 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb800c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb800c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:49.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:49.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.593+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request con 0x7f2bb8003be0 auth_method 0 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb80062e0 auth_method 0 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb800c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb800c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:49:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.797+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:49:49.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122182123 0 0) 0x7f2bcc09f490 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 10 monclient(hunting): _negotiate 2024-09-16T10:49:49.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 10 monclient(hunting): my global_id is 4098 2024-09-16T10:49:49.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.841+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb40035d0 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1545703474 0 0) 0x7f2bb40035d0 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb800c940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb800c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 msgr2=0x7f2bb80087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 --2- 172.21.15.5:0/2940672002 >> v2:172.21.15.143:3300/0 conn(0x7f2bb80062e0 0x7f2bb80087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _finish_hunting 0 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb800df30 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _finish_auth 0 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _check_auth_tickets 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f2bbc016790 con 0x7f2bcc0a14e0 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 5 monclient: authenticate success, global_id 4098 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:49:49.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: dump: 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:49.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 >> v1:172.21.15.5:6789/0 conn(0x7f2bcc0a14e0 legacy=0x7f2bb800b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb4003a90 0x7f2bb4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): picked mon.a con 0x7f2bb4003a90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb4004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): picked mon.c con 0x7f2bb8003be0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb4004db0 0x7f2bb4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): picked mon.b con 0x7f2bb4004db0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc073c20 con 0x7f2bb4003a90 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8008e60 con 0x7f2bb4004db0 2024-09-16T10:49:49.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb400bc90 con 0x7f2bb8003be0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb4004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bb8003be0 auth_method 0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb4004db0 0x7f2bb4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request con 0x7f2bb4004db0 auth_method 0 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb4003a90 0x7f2bb4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bb4003a90 auth_method 0 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 msgr2=0x7f2bb4004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb4004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb4003a90 msgr2=0x7f2bb4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb4003a90 0x7f2bb4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient: _finish_hunting 0 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 monclient: found mon.b 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40035d0 con 0x7f2bb4004db0 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient: _finish_auth 0 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 10 monclient: _check_auth_tickets 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:49.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb4003a90 0x7f2bb4003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb8003be0 0x7f2bb4004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb4004db0 0x7f2bb4005230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc06d5a0 tx=0x7f2bb40058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47087ebb614fa283 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4001dc0 con 0x7f2bb4004db0 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: dump: 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.861 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc400d070 con 0x7f2bb4004db0 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bd29e8700 4 set_mon_vals no callback set 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 1 -- 172.21.15.5:0/2940672002 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40077e0 con 0x7f2bb4004db0 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.845+0000 7f2bcaffd700 10 monclient: dump: 2024-09-16T10:49:49.862 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.863 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: shutdown 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- 172.21.15.5:0/2940672002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb4004db0 msgr2=0x7f2bb4005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 --2- 172.21.15.5:0/2940672002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb4004db0 0x7f2bb4005230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc06d5a0 tx=0x7f2bb40058a0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 1 -- 172.21.15.5:0/2940672002 reap_dead start 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- 172.21.15.5:0/2940672002 shutdown_connections 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- 172.21.15.5:0/2940672002 >> 172.21.15.5:0/2940672002 conn(0x7f2bcc0a20a0 msgr2=0x7f2bcc0a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- 172.21.15.5:0/2940672002 shutdown_connections 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- 172.21.15.5:0/2940672002 wait complete. 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: build_initial_monmap 2024-09-16T10:49:49.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: monmap: 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 librados: starting msgr at 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 librados: starting objecter 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 Processor -- start 2024-09-16T10:49:49.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- start start 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 librados: setting wanted keys 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 librados: calling monclient init 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: init 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bcc0985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-b con 0x7f2bcc1cacc0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-c con 0x7f2bcc071650 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bcc06f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): picked mon.noname-a con 0x7f2bcc06f020 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc070f10 con 0x7f2bcc06f020 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc08ba10 con 0x7f2bcc1cacc0 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc08bcf0 con 0x7f2bcc071650 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd29e8700 10 monclient(hunting): authenticate will time out at 848.430873s 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38590/0 (socket says 172.21.15.5:38590) 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 learned_addr learned my addr 172.21.15.5:0/540921376 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc071650 auth_method 0 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bcc06f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request con 0x7f2bcc06f020 auth_method 0 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bcc0985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bcc1cacc0 auth_method 0 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 msgr2=0x7f2bcc0985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bcc0985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 msgr2=0x7f2bcc06f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bcc06f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 monclient: found mon.noname-c 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:49.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.849+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc070b30 con 0x7f2bcc071650 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc019cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 client.?.objecter ms_handle_connect 0x7f2bcc071650 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc01e050 con 0x7f2bcc071650 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: dump: 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.877 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: _finish_auth 0 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 5 monclient: authenticate success, global_id 4100 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc01f040 con 0x7f2bcc071650 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _renew_subs 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc015c10 con 0x7f2bcc071650 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:49.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc071b70 con 0x7f2bcc071650 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bca7fc700 4 set_mon_vals no callback set 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: dump: 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.879 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _renew_subs 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc06fa10 con 0x7f2bcc071650 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 1 librados: init done 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f2bbc004320 con 0x7f2bcc071650 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2bbc02c7a0 con 0x7f2bcc071650 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 client.4100.objecter ms_dispatch 0x7f2bcc000fd0 osd_map(1..1 src has 1..1) v4 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 20 client.4100.objecter dump_active .. 0 homeless 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:49.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc09a7b0 con 0x7f2bcc071650 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 msgr2=0x7f2bcc098af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 msgr2=0x7f2bcc098af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc019cf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc019cf0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 msgr2=0x7f2bcc098af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bcc071650 0x7f2bcc098af0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bb8007840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): picked mon.a con 0x7f2bcc06f020 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:49.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bb8008030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): picked mon.b con 0x7f2bcc1cacc0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb800d0f0 0x7f2bb800d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): picked mon.c con 0x7f2bb800d0f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8010150 con 0x7f2bcc06f020 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8010290 con 0x7f2bcc1cacc0 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80103d0 con 0x7f2bb800d0f0 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bb8007840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request con 0x7f2bcc06f020 auth_method 0 2024-09-16T10:49:49.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb800d0f0 0x7f2bb800d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bb8008030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request con 0x7f2bb800d0f0 auth_method 0 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request con 0x7f2bcc1cacc0 auth_method 0 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:49.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bd0f85700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient(hunting): handle_auth_done global_id 4100 payload 293 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb800d0f0 msgr2=0x7f2bb800d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb800d0f0 0x7f2bb800d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 msgr2=0x7f2bb8008030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bb8008030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _finish_hunting 0 2024-09-16T10:49:49.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 monclient: found mon.a 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f2bb8010540 con 0x7f2bcc06f020 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc000f040 con 0x7f2bcc06f020 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _finish_auth 0 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 10 monclient: _check_auth_tickets 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 client.4100.objecter ms_handle_connect 0x7f2bcc06f020 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 client.4100.objecter resend_mon_ops 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bcbfff700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bb8007840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00046b0 tx=0x7f2bbc01b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:49.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc001d070 con 0x7f2bcc06f020 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 10 monclient: dump: 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:49.886 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.853+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0006880 con 0x7f2bcc06f020 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f2bc0020070 con 0x7f2bcc06f020 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc9ffb700 4 set_mon_vals no callback set 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f2bcc134470 con 0x7f2bcc06f020 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:49.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.857+0000 7f2bc8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:49.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.933+0000 7f2bd29e8700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:49:49.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.933+0000 7f2bd29e8700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:49:49.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.933+0000 7f2bd29e8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:49.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.933+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2bcc066920 con 0x7f2bcc06f020 2024-09-16T10:49:49.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bc8ff9700 1 -- 172.21.15.5:0/540921376 <== mon.0 v2:172.21.15.5:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2bc00076c0 con 0x7f2bcc06f020 2024-09-16T10:49:49.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bc8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T10:49:49.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bc8ff9700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T10:49:49.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 10 librados: watch_flush enter 2024-09-16T10:49:49.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 10 librados: watch_flush exit 2024-09-16T10:49:49.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 20 client.4100.objecter shutdown clearing up homeless session... 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 10 client.4100.objecter successfully canceled tick 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 10 monclient: shutdown 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 msgr2=0x7f2bb8007840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bb8007840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00046b0 tx=0x7f2bbc01b040 comp rx=0 tx=0).stop 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 shutdown_connections 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2bb800d0f0 0x7f2bb800d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bcc1cacc0 0x7f2bb8008030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 --2- 172.21.15.5:0/540921376 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bcc06f020 0x7f2bb8007840 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 >> 172.21.15.5:0/540921376 conn(0x7f2bcc0a20a0 msgr2=0x7f2bcc09c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 shutdown_connections 2024-09-16T10:49:49.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 -- 172.21.15.5:0/540921376 wait complete. 2024-09-16T10:49:49.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:49.937+0000 7f2bd29e8700 1 librados: shutdown 2024-09-16T10:49:49.948 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-09-16T10:49:49.962 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T10:49:49.962 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T10:49:49.962 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:49.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T10:49:49.977 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:49.977 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T10:49:49.992 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-16T10:49:49.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T10:49:50.008 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:50.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T10:49:50.025 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:50.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T10:49:50.078 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:50.074+0000 7fed798612c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T10:49:50.083 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:49:50.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T10:49:50.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 208764a9-c9f7-49df-9201-b8a2b31be29b 0 2024-09-16T10:49:50.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: get_monmap_and_config 2024-09-16T10:49:50.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.268 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: monmap: 2024-09-16T10:49:50.268 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.268 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:50.203896+0000 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:50.203896+0000 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:50.269 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 Processor -- start 2024-09-16T10:49:50.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 -- start start 2024-09-16T10:49:50.279 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: init 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc0a38d0 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1fc0a21c0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 0x7fe1fc132910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc0a25d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1fc0a3cb0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc0a29e0 con 0x7fe1fc0a38d0 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.280 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0a35a0 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe202ac0700 10 monclient(hunting): authenticate will time out at 850.623211s 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2012bd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe1fc0a21c0 0x7fe1fc132200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41718/0 (socket says 172.21.15.143:41718) 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2012bd700 1 -- 172.21.15.143:0/2038311212 learned_addr learned my addr 172.21.15.143:0/2038311212 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): get_auth_request con 0x7fe1fc0a38d0 auth_method 0 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.281 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 1 --2- 172.21.15.143:0/2038311212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 0x7fe1fc132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): get_auth_request con 0x7fe1fc0a25d0 auth_method 0 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 1 -- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 msgr2=0x7fe1fc135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 1 -- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 msgr2=0x7fe1fc135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 1 --2- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe201abe700 1 --2- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475809962 0 0) 0x7fe1fc0a3cb0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.282 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.198+0000 7fe2022bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient(hunting): _negotiate 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient(hunting): my global_id is 4108 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e80035d0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892092900 0 0) 0x7fe1e80035d0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 -- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 msgr2=0x7fe1fc135bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 --2- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.283 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 -- 172.21.15.143:0/2038311212 >> v1:172.21.15.143:6789/0 conn(0x7fe1fc0a21c0 legacy=0x7fe1fc132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 -- 172.21.15.143:0/2038311212 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc09f9e0 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe2022bf700 1 --2- 172.21.15.143:0/2038311212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 0x7fe1fc132910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009850 tx=0x7fe1f0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3206314da142e98 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40019f0 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.284 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: dump: 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 5 monclient: authenticate success, global_id 4106 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f0004030 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 1 -- 172.21.15.143:0/2038311212 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00043a0 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 4 set_mon_vals no callback set 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe200abc700 10 monclient: dump: 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.286 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 10 monclient: shutdown 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- 172.21.15.143:0/2038311212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 msgr2=0x7fe1fc132910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 --2- 172.21.15.143:0/2038311212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 0x7fe1fc132910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009850 tx=0x7fe1f0009b60 comp rx=0 tx=0).stop 2024-09-16T10:49:50.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- 172.21.15.143:0/2038311212 shutdown_connections 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 --2- 172.21.15.143:0/2038311212 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a25d0 0x7fe1fc132910 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 --2- 172.21.15.143:0/2038311212 >> v2:172.21.15.143:3300/0 conn(0x7fe1fc0a38d0 0x7fe1fc135bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- 172.21.15.143:0/2038311212 >> 172.21.15.143:0/2038311212 conn(0x7fe1fc0a8440 msgr2=0x7fe1fc0a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- 172.21.15.143:0/2038311212 shutdown_connections 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- 172.21.15.143:0/2038311212 wait complete. 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 10 monclient: monmap: 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:50.288 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 librados: starting msgr at 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 librados: starting objecter 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 Processor -- start 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 -- start start 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 librados: setting wanted keys 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 1 librados: calling monclient init 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.202+0000 7fe202ac0700 10 monclient: init 2024-09-16T10:49:50.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc0a21c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1fc0a25d0 0x7fe1fc1cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-a con 0x7fe1fc0a25d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a38d0 0x7fe1fc06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc0a38d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc11ead0 con 0x7fe1fc0a25d0 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0a1520 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc09f630 con 0x7fe1fc0a38d0 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient(hunting): authenticate will time out at 850.628782s 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50720/0 (socket says 172.21.15.143:50720) 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 -- 172.21.15.143:0/1070593976 learned_addr learned my addr 172.21.15.143:0/1070593976 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): get_auth_request con 0x7fe1fc0a21c0 auth_method 0 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2012bd700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1fc0a25d0 0x7fe1fc1cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2022bf700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a38d0 0x7fe1fc06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2012bd700 10 monclient(hunting): get_auth_request con 0x7fe1fc0a25d0 auth_method 0 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2012bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2012bd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2012bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2022bf700 10 monclient(hunting): get_auth_request con 0x7fe1fc0a38d0 auth_method 0 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2022bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.292 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2022bf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe2022bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 -- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a38d0 msgr2=0x7fe1fc06f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a38d0 0x7fe1fc06f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 -- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1fc0a25d0 msgr2=0x7fe1fc1cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1fc0a25d0 0x7fe1fc1cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 -- 172.21.15.143:0/1070593976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00094b0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe201abe700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e8d0 tx=0x7fe1f800ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.293 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 client.?.objecter ms_handle_connect 0x7fe1fc0a21c0 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800eed0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: dump: 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.294 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 5 monclient: authenticate success, global_id 4111 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f800c850 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.295 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _renew_subs 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc0a1f40 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800cbc0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1effff700 4 set_mon_vals no callback set 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 monclient: dump: 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.296 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _renew_subs 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc11ff80 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 librados: init done 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe1f8005d40 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe1f80137b0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 10 client.4111.objecter ms_dispatch 0x7fe1fc000fd0 osd_map(2..2 src has 1..2) v4 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe1ee7fc700 20 client.4111.objecter dump_active .. 0 homeless 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.206+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc135bc0 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.210+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fe1f8021060 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.210+0000 7fe1ee7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.210+0000 7fe1ee7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:50.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.282+0000 7fe202ac0700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "208764a9-c9f7-49df-9201-b8a2b31be29b", "id": 0}] 2024-09-16T10:49:50.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.282+0000 7fe202ac0700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "208764a9-c9f7-49df-9201-b8a2b31be29b", "id": 0}] 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.282+0000 7fe202ac0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.282+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "208764a9-c9f7-49df-9201-b8a2b31be29b", "id": 0} v 0) v1 -- 0x7fe1fc120320 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe1ee7fc700 1 -- 172.21.15.143:0/1070593976 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "208764a9-c9f7-49df-9201-b8a2b31be29b", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe1f8017070 con 0x7fe1fc0a21c0 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe1ee7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "208764a9-c9f7-49df-9201-b8a2b31be29b", "id": 0}] 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe1ee7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:50.300 INFO:teuthology.orchestra.run.smithi143.stdout:0 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 10 librados: watch_flush enter 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 10 librados: watch_flush exit 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 20 client.4111.objecter shutdown clearing up homeless session... 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 10 client.4111.objecter successfully canceled tick 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 10 monclient: shutdown 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 msgr2=0x7fe1fc1ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800e8d0 tx=0x7fe1f800ec90 comp rx=0 tx=0).stop 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 shutdown_connections 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe1fc0a38d0 0x7fe1fc06f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1fc0a25d0 0x7fe1fc1cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 --2- 172.21.15.143:0/1070593976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1fc0a21c0 0x7fe1fc1ca880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 >> 172.21.15.143:0/1070593976 conn(0x7fe1fc0a8440 msgr2=0x7fe1fc0a6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 shutdown_connections 2024-09-16T10:49:50.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 -- 172.21.15.143:0/1070593976 wait complete. 2024-09-16T10:49:50.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.290+0000 7fe202ac0700 1 librados: shutdown 2024-09-16T10:49:50.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 87c369a2-92fc-4740-b6d8-9ab060feea0c 1 2024-09-16T10:49:50.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: get_monmap_and_config 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: monmap: 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:50.422189+0000 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:50.422189+0000 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:50.422 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:50.423 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:50.423 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 Processor -- start 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 -- start start 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: init 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T10:49:50.431 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a0135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-b con 0x7f40a00a38d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40a00a21c0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40a00a25a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a00a3cb0 con 0x7f40a00a21c0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a0133020 con 0x7f40a00a25a0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00a29f0 con 0x7f40a00a38d0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient(hunting): authenticate will time out at 850.841493s 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f40a00a25a0 0x7f40a0132910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41728/0 (socket says 172.21.15.143:41728) 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409ffff700 1 -- 172.21.15.143:0/204870838 learned_addr learned my addr 172.21.15.143:0/204870838 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 --2- 172.21.15.143:0/204870838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a0135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.432 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): get_auth_request con 0x7f40a00a38d0 auth_method 0 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411762965 0 0) 0x7f40a0133020 con 0x7f40a00a25a0 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient(hunting): _negotiate 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient(hunting): my global_id is 4114 2024-09-16T10:49:50.433 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c0035d0 con 0x7f40a00a25a0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 -- 172.21.15.143:0/204870838 >> v1:172.21.15.143:6789/0 conn(0x7f40a00a25a0 legacy=0x7f40a0132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413367599 0 0) 0x7f40a00a3cb0 con 0x7f40a00a21c0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 -- 172.21.15.143:0/204870838 >> v1:172.21.15.5:6789/0 conn(0x7f40a00a21c0 legacy=0x7f40a0132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 -- 172.21.15.143:0/204870838 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a009f9e0 con 0x7f40a00a38d0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409f7fe700 1 --2- 172.21.15.143:0/204870838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a0135bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4098009dc0 tx=0x7f409800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fcc587be25ac2c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.434 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800e070 con 0x7f40a00a38d0 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: dump: 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.435 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 5 monclient: authenticate success, global_id 4112 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409800bb20 con 0x7f40a00a38d0 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f40a48ae700 4 set_mon_vals no callback set 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 1 -- 172.21.15.143:0/204870838 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800be90 con 0x7f40a00a38d0 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.436 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.418+0000 7f409e7fc700 10 monclient: dump: 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.437 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: shutdown 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/204870838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 msgr2=0x7f40a0135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 --2- 172.21.15.143:0/204870838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a0135bc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4098009dc0 tx=0x7f409800b330 comp rx=0 tx=0).stop 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/204870838 shutdown_connections 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 --2- 172.21.15.143:0/204870838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a0135bc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/204870838 >> 172.21.15.143:0/204870838 conn(0x7f40a00a8440 msgr2=0x7f40a00a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/204870838 shutdown_connections 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/204870838 wait complete. 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: monmap: 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.438 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 librados: starting msgr at 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 librados: starting objecter 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 Processor -- start 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- start start 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 librados: setting wanted keys 2024-09-16T10:49:50.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 librados: calling monclient init 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: init 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 0x7f40a006f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-a con 0x7f40a00a21c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-b con 0x7f40a00a25a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a006fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): picked mon.noname-c con 0x7f40a00a38d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0137650 con 0x7f40a00a21c0 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.440 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0137b30 con 0x7f40a00a25a0 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0137e10 con 0x7f40a00a38d0 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient(hunting): authenticate will time out at 850.846331s 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50732/0 (socket says 172.21.15.143:50732) 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 -- 172.21.15.143:0/3720488088 learned_addr learned my addr 172.21.15.143:0/3720488088 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): get_auth_request con 0x7f40a00a25a0 auth_method 0 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.441 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 0x7f40a006f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 10 monclient(hunting): get_auth_request con 0x7f40a00a21c0 auth_method 0 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409ffff700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a006fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409ffff700 10 monclient(hunting): get_auth_request con 0x7f40a00a38d0 auth_method 0 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-16T10:49:50.442 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 -- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 msgr2=0x7f40a006fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a006fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 -- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 msgr2=0x7f40a006f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 0x7f40a006f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 -- 172.21.15.143:0/3720488088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098009a70 con 0x7f40a00a25a0 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409f7fe700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 0x7f40a006f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409effd700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f40880011e0 tx=0x7f408800cae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 client.?.objecter ms_handle_connect 0x7f40a00a25a0 2024-09-16T10:49:50.443 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880054e0 con 0x7f40a00a25a0 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: dump: 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.444 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 5 monclient: authenticate success, global_id 4117 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40880036a0 con 0x7f40a00a25a0 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _renew_subs 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4088003a50 con 0x7f40a00a25a0 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a0137850 con 0x7f40a00a25a0 2024-09-16T10:49:50.445 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f409d7fa700 4 set_mon_vals no callback set 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 monclient: dump: 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.446 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _renew_subs 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a00704f0 con 0x7f40a00a25a0 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f40a48ae700 1 librados: init done 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f408801c440 con 0x7f40a00a25a0 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f40880127b0 con 0x7f40a00a25a0 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 10 client.4117.objecter ms_dispatch 0x7f40a0000fd0 osd_map(3..3 src has 1..3) v4 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.422+0000 7f4097fff700 20 client.4117.objecter dump_active .. 0 homeless 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.426+0000 7f40a48ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.426+0000 7f40a48ae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.447 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.426+0000 7f40a48ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.426+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a013e690 con 0x7f40a00a25a0 2024-09-16T10:49:50.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.430+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f40a00704f0 con 0x7f40a00a25a0 2024-09-16T10:49:50.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.430+0000 7f4097fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.448 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.430+0000 7f4097fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.506+0000 7f40a48ae700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "87c369a2-92fc-4740-b6d8-9ab060feea0c", "id": 1}] 2024-09-16T10:49:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.506+0000 7f40a48ae700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "87c369a2-92fc-4740-b6d8-9ab060feea0c", "id": 1}] 2024-09-16T10:49:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.506+0000 7f40a48ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.506+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87c369a2-92fc-4740-b6d8-9ab060feea0c", "id": 1} v 0) v1 -- 0x7f40a0135bc0 con 0x7f40a00a25a0 2024-09-16T10:49:50.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.510+0000 7f4097fff700 1 -- 172.21.15.143:0/3720488088 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87c369a2-92fc-4740-b6d8-9ab060feea0c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4088017070 con 0x7f40a00a25a0 2024-09-16T10:49:50.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.510+0000 7f4097fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "87c369a2-92fc-4740-b6d8-9ab060feea0c", "id": 1}] 2024-09-16T10:49:50.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.510+0000 7f4097fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:50.518 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-09-16T10:49:50.519 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 10 librados: watch_flush enter 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 10 librados: watch_flush exit 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 10 client.4117.objecter successfully canceled tick 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 10 monclient: shutdown 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 msgr2=0x7f40a006f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f40880011e0 tx=0x7f408800cae0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 shutdown_connections 2024-09-16T10:49:50.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40a00a38d0 0x7f40a006fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40a00a25a0 0x7f40a006f890 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 --2- 172.21.15.143:0/3720488088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40a00a21c0 0x7f40a006f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 >> 172.21.15.143:0/3720488088 conn(0x7f40a00a8440 msgr2=0x7f40a00a62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 shutdown_connections 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 -- 172.21.15.143:0/3720488088 wait complete. 2024-09-16T10:49:50.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.514+0000 7f40a48ae700 1 librados: shutdown 2024-09-16T10:49:50.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 909be8cc-8343-47b5-aa24-41ad442b8837 2 2024-09-16T10:49:50.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 10 monclient: get_monmap_and_config 2024-09-16T10:49:50.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 10 monclient: monmap: 2024-09-16T10:49:50.640 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:50.640839+0000 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:50.640839+0000 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:50.641 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 1 Processor -- start 2024-09-16T10:49:50.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 1 -- start start 2024-09-16T10:49:50.646 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.634+0000 7f48e9bde700 10 monclient: init 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48e40a2dd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e413b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-b con 0x7f48e40a38c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48e40a3ca0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e40a4a10 con 0x7f48e40a2dd0 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e409f150 con 0x7f48e40a3ca0 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e40a4d60 con 0x7f48e40a38c0 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient(hunting): authenticate will time out at 851.060270s 2024-09-16T10:49:50.647 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e93dd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f48e40a3ca0 0x7f48e413d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59820/0 (socket says 172.21.15.143:59820) 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e93dd700 1 -- 172.21.15.143:0/1565834546 learned_addr learned my addr 172.21.15.143:0/1565834546 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 --2- 172.21.15.143:0/1565834546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e413b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): get_auth_request con 0x7f48e40a38c0 auth_method 0 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641274550 0 0) 0x7f48e409f150 con 0x7f48e40a3ca0 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient(hunting): _negotiate 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.648 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient(hunting): my global_id is 4120 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d40035d0 con 0x7f48e40a3ca0 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 -- 172.21.15.143:0/1565834546 >> v1:172.21.15.143:6789/0 conn(0x7f48e40a3ca0 legacy=0x7f48e413d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345510399 0 0) 0x7f48e40a4a10 con 0x7f48e40a2dd0 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 -- 172.21.15.143:0/1565834546 >> v1:172.21.15.5:6789/0 conn(0x7f48e40a2dd0 legacy=0x7f48e40a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 -- 172.21.15.143:0/1565834546 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e409f390 con 0x7f48e40a38c0 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:50.649 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e3fff700 1 --2- 172.21.15.143:0/1565834546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e413b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009880 tx=0x7f48d0009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3682f6b2c643b25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0011830 con 0x7f48e40a38c0 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: dump: 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.650 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 5 monclient: authenticate success, global_id 4118 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0011990 con 0x7f48e40a38c0 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 4 set_mon_vals no callback set 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 1 -- 172.21.15.143:0/1565834546 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0011d20 con 0x7f48e40a38c0 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e37fe700 10 monclient: dump: 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.651 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 10 monclient: shutdown 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- 172.21.15.143:0/1565834546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 msgr2=0x7f48e413b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 --2- 172.21.15.143:0/1565834546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e413b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f48d0009880 tx=0x7f48d0009b90 comp rx=0 tx=0).stop 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- 172.21.15.143:0/1565834546 shutdown_connections 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 --2- 172.21.15.143:0/1565834546 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e413b050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- 172.21.15.143:0/1565834546 >> 172.21.15.143:0/1565834546 conn(0x7f48e40a4ed0 msgr2=0x7f48e40a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- 172.21.15.143:0/1565834546 shutdown_connections 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.638+0000 7f48e9bde700 1 -- 172.21.15.143:0/1565834546 wait complete. 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: monmap: 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:50.653 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 librados: starting msgr at 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 librados: starting objecter 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 Processor -- start 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- start start 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 librados: setting wanted keys 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 librados: calling monclient init 2024-09-16T10:49:50.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: init 2024-09-16T10:49:50.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:49:50.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-b con 0x7f48e40a2dd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e4134fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-c con 0x7f48e40a38c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 0x7f48e4135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): picked mon.noname-a con 0x7f48e40a3ca0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e41ca7c0 con 0x7f48e40a3ca0 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e40a78e0 con 0x7f48e40a2dd0 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e409f5c0 con 0x7f48e40a38c0 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient(hunting): authenticate will time out at 851.066404s 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56980/0 (socket says 172.21.15.143:56980) 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 -- 172.21.15.143:0/3890326389 learned_addr learned my addr 172.21.15.143:0/3890326389 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): get_auth_request con 0x7f48e40a2dd0 auth_method 0 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e3fff700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e4134fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.660 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e93dd700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 0x7f48e4135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 -- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 msgr2=0x7f48e4134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e4134fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 -- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 msgr2=0x7f48e4135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 0x7f48e4135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 -- 172.21.15.143:0/3890326389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d00094b0 con 0x7f48e40a2dd0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e93dd700 10 monclient: get_auth_request con 0x7f48e40a3ca0 auth_method 0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e93dd700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 0x7f48e4135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e8bdc700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c750 tx=0x7f48d800cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.661 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 client.?.objecter ms_handle_connect 0x7f48e40a2dd0 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d80054e0 con 0x7f48e40a2dd0 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: dump: 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.662 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 5 monclient: authenticate success, global_id 4123 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _renew_subs 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d80036a0 con 0x7f48e40a2dd0 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e4135b70 con 0x7f48e40a2dd0 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.663 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8003a50 con 0x7f48e40a2dd0 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 monclient: dump: 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.664 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e2ffd700 4 set_mon_vals no callback set 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _renew_subs 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e41249a0 con 0x7f48e40a2dd0 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e9bde700 1 librados: init done 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f48d801c440 con 0x7f48e40a2dd0 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f48d8012820 con 0x7f48e40a2dd0 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 10 client.4123.objecter ms_dispatch 0x7f48e4000fd0 osd_map(4..4 src has 1..4) v4 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.642+0000 7f48e17fa700 20 client.4123.objecter dump_active .. 0 homeless 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e9bde700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.665 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e9bde700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e9bde700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e409f150 con 0x7f48e40a2dd0 2024-09-16T10:49:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f48d8017070 con 0x7f48e40a2dd0 2024-09-16T10:49:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.646+0000 7f48e17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:50.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.722+0000 7f48e9bde700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "909be8cc-8343-47b5-aa24-41ad442b8837", "id": 2}] 2024-09-16T10:49:50.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.722+0000 7f48e9bde700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "909be8cc-8343-47b5-aa24-41ad442b8837", "id": 2}] 2024-09-16T10:49:50.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.722+0000 7f48e9bde700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.726 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.722+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "909be8cc-8343-47b5-aa24-41ad442b8837", "id": 2} v 0) v1 -- 0x7f48e4122f90 con 0x7f48e40a2dd0 2024-09-16T10:49:50.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.726+0000 7f48e17fa700 1 -- 172.21.15.143:0/3890326389 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "909be8cc-8343-47b5-aa24-41ad442b8837", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f48d8012590 con 0x7f48e40a2dd0 2024-09-16T10:49:50.732 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.726+0000 7f48e17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "909be8cc-8343-47b5-aa24-41ad442b8837", "id": 2}] 2024-09-16T10:49:50.733 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.726+0000 7f48e17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:50.733 INFO:teuthology.orchestra.run.smithi143.stdout:2 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 10 librados: watch_flush enter 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 10 librados: watch_flush exit 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 10 client.4123.objecter successfully canceled tick 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 10 monclient: shutdown 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.734 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 msgr2=0x7f48e41ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c750 tx=0x7f48d800cb10 comp rx=0 tx=0).stop 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 shutdown_connections 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48e40a3ca0 0x7f48e4135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f48e40a38c0 0x7f48e4134fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 --2- 172.21.15.143:0/3890326389 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48e40a2dd0 0x7f48e41ca280 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 >> 172.21.15.143:0/3890326389 conn(0x7f48e40a4ed0 msgr2=0x7f48e4135da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 shutdown_connections 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 -- 172.21.15.143:0/3890326389 wait complete. 2024-09-16T10:49:50.735 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.730+0000 7f48e9bde700 1 librados: shutdown 2024-09-16T10:49:50.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new bdaac9ae-7e26-471d-a9ed-95855c346e82 3 2024-09-16T10:49:50.830 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:50.826+0000 7fed798612c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T10:49:50.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: get_monmap_and_config 2024-09-16T10:49:50.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.855 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: monmap: 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:50.856212+0000 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:50.856212+0000 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:50.856 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 Processor -- start 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 -- start start 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: init 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 0x7fa9d4135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.865 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-b con 0x7fa9d40a38d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa9d40a21c0 0x7fa9d40a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-a con 0x7fa9d40a21c0 addr v2:172.21.15.5:3300/0 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9d4132200 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40a3cb0 con 0x7fa9d4132200 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d40a35a0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d409f9e0 con 0x7fa9d40a38d0 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d9ef7700 10 monclient(hunting): authenticate will time out at 851.275725s 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.850+0000 7fa9d96f6700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa9d4132200 0x7fa9d4132650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59826/0 (socket says 172.21.15.143:59826) 2024-09-16T10:49:50.866 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 0x7fa9d4135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d96f6700 1 -- 172.21.15.143:0/1712142130 learned_addr learned my addr 172.21.15.143:0/1712142130 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): get_auth_request con 0x7fa9d40a38d0 auth_method 0 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 1 --2- 172.21.15.143:0/1712142130 >> v2:172.21.15.5:3300/0 conn(0x7fa9d40a21c0 0x7fa9d40a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): get_auth_request con 0x7fa9d40a21c0 auth_method 0 2024-09-16T10:49:50.867 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 1 -- 172.21.15.143:0/1712142130 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529172213 0 0) 0x7fa9d40a3cb0 con 0x7fa9d4132200 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient(hunting): _negotiate 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient(hunting): my global_id is 4126 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 1 -- 172.21.15.143:0/1712142130 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c40035d0 con 0x7fa9d4132200 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-09-16T10:49:50.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1712142130 >> v2:172.21.15.5:3300/0 conn(0x7fa9d40a21c0 msgr2=0x7fa9d40a25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 --2- 172.21.15.143:0/1712142130 >> v2:172.21.15.5:3300/0 conn(0x7fa9d40a21c0 0x7fa9d40a25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1712142130 >> v1:172.21.15.143:6789/0 conn(0x7fa9d4132200 legacy=0x7fa9d4132650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1712142130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40a4c50 con 0x7fa9d40a38d0 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d8ef5700 1 --2- 172.21.15.143:0/1712142130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 0x7fa9d4135bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009850 tx=0x7fa9c8009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50d2bc21b330dfaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 1 -- 172.21.15.143:0/1712142130 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8011830 con 0x7fa9d40a38d0 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: dump: 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 5 monclient: authenticate success, global_id 4121 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 1 -- 172.21.15.143:0/1712142130 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c8011990 con 0x7fa9d40a38d0 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 4 set_mon_vals no callback set 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 1 -- 172.21.15.143:0/1712142130 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8011d40 con 0x7fa9d40a38d0 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:50.871 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d37fe700 10 monclient: dump: 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: shutdown 2024-09-16T10:49:50.872 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1712142130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 msgr2=0x7fa9d4135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1712142130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 0x7fa9d4135bc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009850 tx=0x7fa9c8009b60 comp rx=0 tx=0).stop 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1712142130 shutdown_connections 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1712142130 >> v2:172.21.15.5:3300/0 conn(0x7fa9d40a21c0 0x7fa9d40a25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1712142130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d40a38d0 0x7fa9d4135bc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1712142130 >> 172.21.15.143:0/1712142130 conn(0x7fa9d40a8440 msgr2=0x7fa9d40a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1712142130 shutdown_connections 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1712142130 wait complete. 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: build_initial_monmap 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: monmap: 2024-09-16T10:49:50.873 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 librados: starting msgr at 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 librados: starting objecter 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 Processor -- start 2024-09-16T10:49:50.874 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- start start 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 librados: setting wanted keys 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 librados: calling monclient init 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: init 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-b con 0x7fa9d40a21c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 0x7fa9d41cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-a con 0x7fa9d40a38d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 0x7fa9d406c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): picked mon.noname-c con 0x7fa9d4132200 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.875 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41cb4e0 con 0x7fa9d40a38d0 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d4131f40 con 0x7fa9d40a21c0 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d409f630 con 0x7fa9d4132200 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.854+0000 7fa9d9ef7700 10 monclient(hunting): authenticate will time out at 851.279895s 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:56984/0 (socket says 172.21.15.143:56984) 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1710787459 learned_addr learned my addr 172.21.15.143:0/1710787459 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): get_auth_request con 0x7fa9d40a21c0 auth_method 0 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.876 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 0x7fa9d406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 0x7fa9d41cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 10 monclient(hunting): get_auth_request con 0x7fa9d40a38d0 auth_method 0 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 10 monclient(hunting): get_auth_request con 0x7fa9d4132200 auth_method 0 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:50.877 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 msgr2=0x7fa9d406c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 0x7fa9d406c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 msgr2=0x7fa9d41cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 0x7fa9d41cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient: _finish_hunting 0 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 monclient: found mon.noname-b 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 -- 172.21.15.143:0/1710787459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80094b0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d96f6700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 0x7fa9d406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d3fff700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 0x7fa9d41cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:49:50.878 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d8ef5700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80090b0 tx=0x7fa9c8004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 client.?.objecter ms_handle_connect 0x7fa9d40a21c0 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8004480 con 0x7fa9d40a21c0 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: dump: 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.879 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: _finish_auth 0 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: _check_auth_tickets 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 5 monclient: authenticate success, global_id 4129 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c801f070 con 0x7fa9d40a21c0 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _renew_subs 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d406fab0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8003900 con 0x7fa9d40a21c0 2024-09-16T10:49:50.880 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 monclient: dump: 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:50.881 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _renew_subs 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d4070ae0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 1 librados: init done 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d2ffd700 4 set_mon_vals no callback set 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fa9c8021030 con 0x7fa9d40a21c0 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fa9c8024430 con 0x7fa9d40a21c0 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 10 client.4129.objecter ms_dispatch 0x7fa9d4000fd0 osd_map(5..5 src has 1..5) v4 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d17fa700 20 client.4129.objecter dump_active .. 0 homeless 2024-09-16T10:49:50.882 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.858+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d406ffd0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.862+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fa9d406ffd0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.862+0000 7fa9d17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:50.883 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.862+0000 7fa9d17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:50.890 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:50.886+0000 7fed798612c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T10:49:50.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.938+0000 7fa9d9ef7700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bdaac9ae-7e26-471d-a9ed-95855c346e82", "id": 3}] 2024-09-16T10:49:50.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.938+0000 7fa9d9ef7700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bdaac9ae-7e26-471d-a9ed-95855c346e82", "id": 3}] 2024-09-16T10:49:50.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.938+0000 7fa9d9ef7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:50.942 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.938+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bdaac9ae-7e26-471d-a9ed-95855c346e82", "id": 3} v 0) v1 -- 0x7fa9d40a49a0 con 0x7fa9d40a21c0 2024-09-16T10:49:50.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.942+0000 7fa9d17fa700 1 -- 172.21.15.143:0/1710787459 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bdaac9ae-7e26-471d-a9ed-95855c346e82", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa9c8015020 con 0x7fa9d40a21c0 2024-09-16T10:49:50.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.942+0000 7fa9d17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bdaac9ae-7e26-471d-a9ed-95855c346e82", "id": 3}] 2024-09-16T10:49:50.948 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.942+0000 7fa9d17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:50.949 INFO:teuthology.orchestra.run.smithi143.stdout:3 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 10 librados: watch_flush enter 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 10 librados: watch_flush exit 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 10 client.4129.objecter successfully canceled tick 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 10 monclient: shutdown 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 msgr2=0x7fa9d41caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80090b0 tx=0x7fa9c8004770 comp rx=0 tx=0).stop 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 shutdown_connections 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa9d4132200 0x7fa9d406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9d40a38d0 0x7fa9d41cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 --2- 172.21.15.143:0/1710787459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9d40a21c0 0x7fa9d41caa60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:50.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 >> 172.21.15.143:0/1710787459 conn(0x7fa9d40a8440 msgr2=0x7fa9d40a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:50.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 shutdown_connections 2024-09-16T10:49:50.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 -- 172.21.15.143:0/1710787459 wait complete. 2024-09-16T10:49:50.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:50.946+0000 7fa9d9ef7700 1 librados: shutdown 2024-09-16T10:49:50.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 4d9fc719-195a-41dc-bb5f-13b662146355 4 2024-09-16T10:49:51.066 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.062+0000 7fed798612c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: get_monmap_and_config 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: monmap: 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.074 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:51.074939+0000 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:51.074939+0000 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:51.075 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 Processor -- start 2024-09-16T10:49:51.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 -- start start 2024-09-16T10:49:51.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: init 2024-09-16T10:49:51.090 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f54800a0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): picked mon.noname-b con 0x7f54800a1c70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54800a2050 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5480138c60 0x7f548013b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): picked mon.noname-a con 0x7f5480138c60 addr v2:172.21.15.5:3300/0 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548009f6c0 con 0x7f54800a2050 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548009dde0 con 0x7f5480138c60 2024-09-16T10:49:51.091 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548009f100 con 0x7f54800a1c70 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 10 monclient(hunting): authenticate will time out at 851.494899s 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547f7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f54800a2050 0x7f54800a2450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59830/0 (socket says 172.21.15.143:59830) 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547f7fe700 1 -- 172.21.15.143:0/743949353 learned_addr learned my addr 172.21.15.143:0/743949353 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f54800a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): get_auth_request con 0x7f54800a1c70 auth_method 0 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.092 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 1 --2- 172.21.15.143:0/743949353 >> v2:172.21.15.5:3300/0 conn(0x7f5480138c60 0x7f548013b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): get_auth_request con 0x7f5480138c60 auth_method 0 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613978825 0 0) 0x7f548009f6c0 con 0x7f54800a2050 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient(hunting): _negotiate 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient(hunting): my global_id is 4132 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54680035d0 con 0x7f54800a2050 2024-09-16T10:49:51.093 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006289063 0 0) 0x7f54680035d0 con 0x7f54800a2050 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 msgr2=0x7f54800a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f54800a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 >> v2:172.21.15.5:3300/0 conn(0x7f5480138c60 msgr2=0x7f548013b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 --2- 172.21.15.143:0/743949353 >> v2:172.21.15.5:3300/0 conn(0x7f5480138c60 0x7f548013b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 monclient: found mon.noname-c-legacy 2024-09-16T10:49:51.094 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548009f3e0 con 0x7f54800a2050 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f5474001670 con 0x7f54800a2050 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 5 monclient: authenticate success, global_id 4132 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f548571e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: dump: 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.095 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 >> v1:172.21.15.143:6789/0 conn(0x7f54800a2050 legacy=0x7f54800a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:49:51.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468003a90 0x7f5468003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.070+0000 7f547effd700 10 monclient(hunting): picked mon.a con 0x7f5468003a90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f5468004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): picked mon.c con 0x7f54800a1c70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5468004db0 0x7f5468005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): picked mon.b con 0x7f5468004db0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548009dde0 con 0x7f5468003a90 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548009f100 con 0x7f5468004db0 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546800bb30 con 0x7f54800a1c70 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5468004db0 0x7f5468005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): get_auth_request con 0x7f5468004db0 auth_method 0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468003a90 0x7f5468003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f5468004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): get_auth_request con 0x7f5468003a90 auth_method 0 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): get_auth_request con 0x7f54800a1c70 auth_method 0 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient(hunting): handle_auth_done global_id 4132 payload 293 2024-09-16T10:49:51.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 -- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 msgr2=0x7f5468004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f54800a1c70 0x7f5468004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 -- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468003a90 msgr2=0x7f5468003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468003a90 0x7f5468003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 monclient: found mon.b 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 -- 172.21.15.143:0/743949353 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680035d0 con 0x7f5468004db0 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5468004db0 0x7f5468005230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5478009f90 tx=0x7f54680058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ab26994239af678 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547800ec30 con 0x7f5468004db0 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: dump: 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.101 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547800f040 con 0x7f5468004db0 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 4 set_mon_vals no callback set 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 1 -- 172.21.15.143:0/743949353 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478014480 con 0x7f5468004db0 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f547effd700 10 monclient: dump: 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.102 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.103 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 10 monclient: shutdown 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- 172.21.15.143:0/743949353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5468004db0 msgr2=0x7f5468005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 --2- 172.21.15.143:0/743949353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5468004db0 0x7f5468005230 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5478009f90 tx=0x7f54680058a0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f5484f1d700 1 -- 172.21.15.143:0/743949353 reap_dead start 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- 172.21.15.143:0/743949353 shutdown_connections 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- 172.21.15.143:0/743949353 >> 172.21.15.143:0/743949353 conn(0x7f54800a6890 msgr2=0x7f54800a2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- 172.21.15.143:0/743949353 shutdown_connections 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- 172.21.15.143:0/743949353 wait complete. 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 10 monclient: monmap: 2024-09-16T10:49:51.104 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 librados: starting msgr at 2024-09-16T10:49:51.105 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 librados: starting objecter 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 Processor -- start 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 -- start start 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 librados: setting wanted keys 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 1 librados: calling monclient init 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.074+0000 7f548571e700 10 monclient: init 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468004db0 0x7f548009b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): picked mon.noname-a con 0x7f5468004db0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): picked mon.noname-b con 0x7f548009b860 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f548009c1c0 0x7f54801d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.106 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): picked mon.noname-c con 0x7f548009c1c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5480131c40 con 0x7f5468004db0 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548006cfa0 con 0x7f548009b860 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548006df20 con 0x7f548009c1c0 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient(hunting): authenticate will time out at 851.500598s 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57006/0 (socket says 172.21.15.143:57006) 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 -- 172.21.15.143:0/30705347 learned_addr learned my addr 172.21.15.143:0/30705347 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): get_auth_request con 0x7f548009b860 auth_method 0 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.107 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468004db0 0x7f548009b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f548009c1c0 0x7f54801d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): get_auth_request con 0x7f5468004db0 auth_method 0 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): get_auth_request con 0x7f548009c1c0 auth_method 0 2024-09-16T10:49:51.108 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f5484f1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 -- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f548009c1c0 msgr2=0x7f54801d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f548009c1c0 0x7f54801d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 -- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468004db0 msgr2=0x7f548009b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468004db0 0x7f548009b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.109 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 monclient: found mon.noname-b 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 -- 172.21.15.143:0/30705347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5480120ea0 con 0x7f548009b860 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547f7fe700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5474007370 tx=0x7f5474007680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 client.?.objecter ms_handle_connect 0x7f548009b860 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474007e60 con 0x7f548009b860 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.110 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: dump: 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.111 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 5 monclient: authenticate success, global_id 4135 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _renew_subs 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54740077a0 con 0x7f548009b860 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548006eaa0 con 0x7f548009b860 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547400faf0 con 0x7f548009b860 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547e7fc700 4 set_mon_vals no callback set 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 monclient: dump: 2024-09-16T10:49:51.112 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.113 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _renew_subs 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548006c3d0 con 0x7f548009b860 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 librados: init done 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5474014450 con 0x7f548009b860 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5474014c10 con 0x7f548009b860 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 10 client.4135.objecter ms_dispatch 0x7f5480000fd0 osd_map(6..6 src has 1..6) v4 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-09-16T10:49:51.114 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f547cff9700 20 client.4135.objecter dump_active .. 0 homeless 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.078+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54800a2450 con 0x7f548009b860 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.082+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f547401a070 con 0x7f548009b860 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.082+0000 7f547cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.115 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.082+0000 7f547cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:51.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.174+0000 7f548571e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4d9fc719-195a-41dc-bb5f-13b662146355", "id": 4}] 2024-09-16T10:49:51.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.174+0000 7f548571e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4d9fc719-195a-41dc-bb5f-13b662146355", "id": 4}] 2024-09-16T10:49:51.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.174+0000 7f548571e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.174+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d9fc719-195a-41dc-bb5f-13b662146355", "id": 4} v 0) v1 -- 0x7f548006e060 con 0x7f548009b860 2024-09-16T10:49:51.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f547cff9700 1 -- 172.21.15.143:0/30705347 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d9fc719-195a-41dc-bb5f-13b662146355", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5474014700 con 0x7f548009b860 2024-09-16T10:49:51.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f547cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4d9fc719-195a-41dc-bb5f-13b662146355", "id": 4}] 2024-09-16T10:49:51.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f547cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:51.186 INFO:teuthology.orchestra.run.smithi143.stdout:4 2024-09-16T10:49:51.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 10 librados: watch_flush enter 2024-09-16T10:49:51.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 10 librados: watch_flush exit 2024-09-16T10:49:51.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 20 client.4135.objecter shutdown clearing up homeless session... 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 10 client.4135.objecter successfully canceled tick 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 10 monclient: shutdown 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 msgr2=0x7f548009bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5474007370 tx=0x7f5474007680 comp rx=0 tx=0).stop 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 shutdown_connections 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f548009c1c0 0x7f54801d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f548009b860 0x7f548009bc80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 --2- 172.21.15.143:0/30705347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5468004db0 0x7f548009b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 >> 172.21.15.143:0/30705347 conn(0x7f54800a6890 msgr2=0x7f5480133e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 shutdown_connections 2024-09-16T10:49:51.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 -- 172.21.15.143:0/30705347 wait complete. 2024-09-16T10:49:51.189 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.182+0000 7f548571e700 1 librados: shutdown 2024-09-16T10:49:51.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new d4673f5b-8a8e-4559-a2a6-dadcc95e28b5 5 2024-09-16T10:49:51.294 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.290+0000 7fed798612c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T10:49:51.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: get_monmap_and_config 2024-09-16T10:49:51.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: monmap: 2024-09-16T10:49:51.310 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:51.310 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:51.310126+0000 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:51.310126+0000 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:51.311 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:51.312 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:51.312 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:51.312 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:51.312 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 Processor -- start 2024-09-16T10:49:51.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 -- start start 2024-09-16T10:49:51.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: init 2024-09-16T10:49:51.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.329 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T10:49:51.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52940a41d0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:51.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52940a0360 0x7f52940a2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-c con 0x7f52940a0360 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:51.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52940a4cc0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:51.330 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940a1d30 con 0x7f52940a41d0 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940a1400 con 0x7f52940a4cc0 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52940a19c0 con 0x7f52940a0360 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient(hunting): authenticate will time out at 851.730370s 2024-09-16T10:49:51.331 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529ab40700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f52940a4cc0 0x7f529413d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59844/0 (socket says 172.21.15.143:59844) 2024-09-16T10:49:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52940a0360 0x7f52940a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529ab40700 1 -- 172.21.15.143:0/30938573 learned_addr learned my addr 172.21.15.143:0/30938573 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T10:49:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): get_auth_request con 0x7f52940a0360 auth_method 0 2024-09-16T10:49:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.332 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.333 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 1 -- 172.21.15.143:0/30938573 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255896703 0 0) 0x7f52940a1400 con 0x7f52940a4cc0 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient(hunting): _negotiate 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient(hunting): my global_id is 4141 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 1 -- 172.21.15.143:0/30938573 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52800035d0 con 0x7f52940a4cc0 2024-09-16T10:49:51.334 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-09-16T10:49:51.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 -- 172.21.15.143:0/30938573 >> v1:172.21.15.143:6789/0 conn(0x7f52940a4cc0 legacy=0x7f529413d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 -- 172.21.15.143:0/30938573 >> v1:172.21.15.5:6789/0 conn(0x7f52940a41d0 legacy=0x7f52940a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 monclient: found mon.noname-c 2024-09-16T10:49:51.335 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 -- 172.21.15.143:0/30938573 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940a16e0 con 0x7f52940a0360 2024-09-16T10:49:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f5299b3e700 1 --2- 172.21.15.143:0/30938573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a0360 0x7f52940a2580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5284009850 tx=0x7f5284009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a06f9bb77cf6ea3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 1 -- 172.21.15.143:0/30938573 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284011830 con 0x7f52940a0360 2024-09-16T10:49:51.336 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:49:51.337 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: dump: 2024-09-16T10:49:51.337 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.337 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.337 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.338 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 5 monclient: authenticate success, global_id 4138 2024-09-16T10:49:51.339 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 1 -- 172.21.15.143:0/30938573 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284011990 con 0x7f52940a0360 2024-09-16T10:49:51.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:51.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:51.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529c5a3700 4 set_mon_vals no callback set 2024-09-16T10:49:51.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 1 -- 172.21.15.143:0/30938573 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5284011d40 con 0x7f52940a0360 2024-09-16T10:49:51.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:51.341 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.306+0000 7f529933d700 10 monclient: dump: 2024-09-16T10:49:51.341 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.342 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.342 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.342 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.342 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.342 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: shutdown 2024-09-16T10:49:51.343 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- 172.21.15.143:0/30938573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a0360 msgr2=0x7f52940a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 --2- 172.21.15.143:0/30938573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a0360 0x7f52940a2580 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5284009850 tx=0x7f5284009b60 comp rx=0 tx=0).stop 2024-09-16T10:49:51.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- 172.21.15.143:0/30938573 shutdown_connections 2024-09-16T10:49:51.344 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 --2- 172.21.15.143:0/30938573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a0360 0x7f52940a2580 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- 172.21.15.143:0/30938573 >> 172.21.15.143:0/30938573 conn(0x7f52940a52c0 msgr2=0x7f52940a56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- 172.21.15.143:0/30938573 shutdown_connections 2024-09-16T10:49:51.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- 172.21.15.143:0/30938573 wait complete. 2024-09-16T10:49:51.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.345 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: monmap: 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.346 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T10:49:51.347 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:51.347 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:51.347 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:51.347 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.347 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 librados: starting msgr at 2024-09-16T10:49:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 librados: starting objecter 2024-09-16T10:49:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 Processor -- start 2024-09-16T10:49:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- start start 2024-09-16T10:49:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 librados: setting wanted keys 2024-09-16T10:49:51.348 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 librados: calling monclient init 2024-09-16T10:49:51.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: init 2024-09-16T10:49:51.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:49:51.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52940a0360 0x7f52941cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.349 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-a con 0x7f52940a0360 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52940a41d0 0x7f52941cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-c con 0x7f52940a41d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): picked mon.noname-b con 0x7f52940a4cc0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.350 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52941395e0 con 0x7f52940a0360 2024-09-16T10:49:51.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529413ca00 con 0x7f52940a4cc0 2024-09-16T10:49:51.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529413cfc0 con 0x7f52940a41d0 2024-09-16T10:49:51.351 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529c5a3700 10 monclient(hunting): authenticate will time out at 851.735130s 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57024/0 (socket says 172.21.15.143:57024) 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 -- 172.21.15.143:0/4098921178 learned_addr learned my addr 172.21.15.143:0/4098921178 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): get_auth_request con 0x7f52940a4cc0 auth_method 0 2024-09-16T10:49:51.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529a33f700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52940a0360 0x7f52941cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529a33f700 10 monclient(hunting): get_auth_request con 0x7f52940a0360 auth_method 0 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529a33f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529a33f700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529a33f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f5299b3e700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52940a41d0 0x7f52941cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-16T10:49:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 -- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52940a41d0 msgr2=0x7f52941cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52940a41d0 0x7f52941cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 -- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52940a0360 msgr2=0x7f52941cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52940a0360 0x7f52941cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 monclient: found mon.noname-b 2024-09-16T10:49:51.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 -- 172.21.15.143:0/4098921178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52840094b0 con 0x7f52940a4cc0 2024-09-16T10:49:51.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f529ab40700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f528c00caa0 tx=0x7f528c00cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f528affd700 10 client.?.objecter ms_handle_connect 0x7f52940a4cc0 2024-09-16T10:49:51.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f528affd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:51.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c00a7b0 con 0x7f52940a4cc0 2024-09-16T10:49:51.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.310+0000 7f528affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: dump: 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.358 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.359 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.359 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.359 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.359 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.359 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.360 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.360 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 5 monclient: authenticate success, global_id 4144 2024-09-16T10:49:51.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _renew_subs 2024-09-16T10:49:51.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c00a910 con 0x7f52940a4cc0 2024-09-16T10:49:51.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529413d2a0 con 0x7f52940a4cc0 2024-09-16T10:49:51.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c00aca0 con 0x7f52940a4cc0 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f5298b3c700 4 set_mon_vals no callback set 2024-09-16T10:49:51.362 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 monclient: dump: 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.363 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T10:49:51.364 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.364 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.364 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.364 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T10:49:51.364 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _renew_subs 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529413da30 con 0x7f52940a4cc0 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 1 librados: init done 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f528c01c440 con 0x7f52940a4cc0 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f528c012980 con 0x7f52940a4cc0 2024-09-16T10:49:51.365 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 10 client.4144.objecter ms_dispatch 0x7f5294000fd0 osd_map(7..7 src has 1..7) v4 2024-09-16T10:49:51.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-09-16T10:49:51.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-09-16T10:49:51.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f528affd700 20 client.4144.objecter dump_active .. 0 homeless 2024-09-16T10:49:51.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.366 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.314+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5294066920 con 0x7f52940a4cc0 2024-09-16T10:49:51.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.318+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f528c017070 con 0x7f52940a4cc0 2024-09-16T10:49:51.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.318+0000 7f528affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.367 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.318+0000 7f528affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:51.368 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.338+0000 7fed798612c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T10:49:51.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.394+0000 7f529c5a3700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d4673f5b-8a8e-4559-a2a6-dadcc95e28b5", "id": 5}] 2024-09-16T10:49:51.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.394+0000 7f529c5a3700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d4673f5b-8a8e-4559-a2a6-dadcc95e28b5", "id": 5}] 2024-09-16T10:49:51.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.394+0000 7f529c5a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:49:51.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.394+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d4673f5b-8a8e-4559-a2a6-dadcc95e28b5", "id": 5} v 0) v1 -- 0x7f52940a2580 con 0x7f52940a4cc0 2024-09-16T10:49:51.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f528affd700 1 -- 172.21.15.143:0/4098921178 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d4673f5b-8a8e-4559-a2a6-dadcc95e28b5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f528c0124d0 con 0x7f52940a4cc0 2024-09-16T10:49:51.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f528affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d4673f5b-8a8e-4559-a2a6-dadcc95e28b5", "id": 5}] 2024-09-16T10:49:51.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f528affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:51.406 INFO:teuthology.orchestra.run.smithi143.stdout:5 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 10 librados: watch_flush enter 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 10 librados: watch_flush exit 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 20 client.4144.objecter shutdown clearing up homeless session... 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 10 client.4144.objecter successfully canceled tick 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 10 monclient: shutdown 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 msgr2=0x7f52941cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f528c00caa0 tx=0x7f528c00cb80 comp rx=0 tx=0).stop 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 shutdown_connections 2024-09-16T10:49:51.408 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52940a4cc0 0x7f52941cb880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f52940a41d0 0x7f52941cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 --2- 172.21.15.143:0/4098921178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f52940a0360 0x7f52941cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 >> 172.21.15.143:0/4098921178 conn(0x7f52940a52c0 msgr2=0x7f52940a7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 shutdown_connections 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 -- 172.21.15.143:0/4098921178 wait complete. 2024-09-16T10:49:51.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T10:49:51.402+0000 7f529c5a3700 1 librados: shutdown 2024-09-16T10:49:51.421 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T10:49:51.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T10:49:51.423 INFO:tasks.ceph.osd.0:Started 2024-09-16T10:49:51.424 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T10:49:51.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T10:49:51.426 INFO:tasks.ceph.osd.1:Started 2024-09-16T10:49:51.426 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T10:49:51.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T10:49:51.428 INFO:tasks.ceph.osd.2:Started 2024-09-16T10:49:51.428 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T10:49:51.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T10:49:51.430 INFO:tasks.ceph.osd.3:Started 2024-09-16T10:49:51.431 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T10:49:51.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T10:49:51.432 INFO:tasks.ceph.osd.4:Started 2024-09-16T10:49:51.432 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T10:49:51.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T10:49:51.434 INFO:tasks.ceph.osd.5:Started 2024-09-16T10:49:51.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:49:51.548 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.542+0000 7fed798612c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: get_monmap_and_config 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: monmap: 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:51.549314+0000 2024-09-16T10:49:51.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:51.549314+0000 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:51.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 Processor -- start 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 -- start start 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: init 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62280a0b50 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6228098bd0 0x7f6228074330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-a con 0x7f6228098bd0 addr v2:172.21.15.5:3300/0 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f622813b2f0 0x7f622813d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-c con 0x7f622813b2f0 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62280a0f30 con 0x7f62280a0b50 2024-09-16T10:49:51.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62280a2b60 con 0x7f6228098bd0 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62280a1370 con 0x7f622813b2f0 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient(hunting): authenticate will time out at 850.126348s 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622c9cd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f62280a0b50 0x7f6228073c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35712/0 (socket says 172.21.15.5:35712) 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622c9cd700 1 -- 172.21.15.5:0/1760768100 learned_addr learned my addr 172.21.15.5:0/1760768100 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 --2- 172.21.15.5:0/1760768100 >> v2:172.21.15.5:3300/0 conn(0x7f6228098bd0 0x7f6228074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): get_auth_request con 0x7f6228098bd0 auth_method 0 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 1 --2- 172.21.15.5:0/1760768100 >> v2:172.21.15.143:3300/0 conn(0x7f622813b2f0 0x7f622813d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): get_auth_request con 0x7f622813b2f0 auth_method 0 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 1 -- 172.21.15.5:0/1760768100 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937550559 0 0) 0x7f62280a0f30 con 0x7f62280a0b50 2024-09-16T10:49:51.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient(hunting): _negotiate 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient(hunting): my global_id is 4164 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 1 -- 172.21.15.5:0/1760768100 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62100035d0 con 0x7f62280a0b50 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient(hunting): handle_auth_done global_id 4161 payload 274 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 -- 172.21.15.5:0/1760768100 >> v2:172.21.15.143:3300/0 conn(0x7f622813b2f0 msgr2=0x7f622813d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 --2- 172.21.15.5:0/1760768100 >> v2:172.21.15.143:3300/0 conn(0x7f622813b2f0 0x7f622813d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 -- 172.21.15.5:0/1760768100 >> v1:172.21.15.5:6789/0 conn(0x7f62280a0b50 legacy=0x7f6228073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 monclient: found mon.noname-a 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 -- 172.21.15.5:0/1760768100 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622806b170 con 0x7f6228098bd0 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f6227fff700 1 --2- 172.21.15.5:0/1760768100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6228098bd0 0x7f6228074330 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f622806d740 tx=0x7f621c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fa36c0985a080b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 1 -- 172.21.15.5:0/1760768100 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c011830 con 0x7f6228098bd0 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: dump: 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.561 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 1 -- 172.21.15.5:0/1760768100 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c011990 con 0x7f6228098bd0 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 5 monclient: authenticate success, global_id 4161 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 1 -- 172.21.15.5:0/1760768100 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c011d20 con 0x7f6228098bd0 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f622ec31700 4 set_mon_vals no callback set 2024-09-16T10:49:51.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.545+0000 7f62277fe700 10 monclient: dump: 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: shutdown 2024-09-16T10:49:51.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- 172.21.15.5:0/1760768100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6228098bd0 msgr2=0x7f6228074330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- 172.21.15.5:0/1760768100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6228098bd0 0x7f6228074330 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f622806d740 tx=0x7f621c009b30 comp rx=0 tx=0).stop 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- 172.21.15.5:0/1760768100 shutdown_connections 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- 172.21.15.5:0/1760768100 >> v2:172.21.15.143:3300/0 conn(0x7f622813b2f0 0x7f622813d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- 172.21.15.5:0/1760768100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6228098bd0 0x7f6228074330 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- 172.21.15.5:0/1760768100 >> 172.21.15.5:0/1760768100 conn(0x7f62280a1c40 msgr2=0x7f62280a2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- 172.21.15.5:0/1760768100 shutdown_connections 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- 172.21.15.5:0/1760768100 wait complete. 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: build_initial_monmap 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: monmap: 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:51.564 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 librados: starting msgr at 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 librados: starting objecter 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 Processor -- start 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- start start 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 librados: setting wanted keys 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 librados: calling monclient init 2024-09-16T10:49:51.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: init 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-c con 0x7f6228098bd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-a con 0x7f62280a0b50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f622813b2f0 0x7f6228091000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): picked mon.noname-b con 0x7f622813b2f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6228094be0 con 0x7f62280a0b50 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6228093070 con 0x7f622813b2f0 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6228094900 con 0x7f6228098bd0 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622ec31700 10 monclient(hunting): authenticate will time out at 850.130799s 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38640/0 (socket says 172.21.15.5:38640) 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 1 -- 172.21.15.5:0/792347496 learned_addr learned my addr 172.21.15.5:0/792347496 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): get_auth_request con 0x7f6228098bd0 auth_method 0 2024-09-16T10:49:51.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35820/0 (socket says 172.21.15.5:35820) 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): get_auth_request con 0x7f62280a0b50 auth_method 0 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f622813b2f0 0x7f6228091000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): get_auth_request con 0x7f622813b2f0 auth_method 0 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:51.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622d1ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient(hunting): handle_auth_done global_id 4167 payload 824 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 -- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 msgr2=0x7f6228097400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 -- 172.21.15.5:0/792347496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f622813b2f0 msgr2=0x7f6228091000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f622813b2f0 0x7f6228091000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient: _finish_hunting 0 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 monclient: found mon.noname-a 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 -- 172.21.15.5:0/792347496 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c0094b0 con 0x7f62280a0b50 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f622c9cd700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 client.?.objecter ms_handle_connect 0x7f62280a0b50 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.549+0000 7f6227fff700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f621c016000 tx=0x7f621c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c004590 con 0x7f62280a0b50 2024-09-16T10:49:51.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: dump: 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: _finish_auth 0 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: _check_auth_tickets 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c0036c0 con 0x7f62280a0b50 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 5 monclient: authenticate success, global_id 4167 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _renew_subs 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6228092b60 con 0x7f62280a0b50 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c003a70 con 0x7f62280a0b50 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 monclient: dump: 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:51.576 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _renew_subs 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f6226ffd700 4 set_mon_vals no callback set 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62280a3e20 con 0x7f62280a0b50 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 1 librados: init done 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f621c021070 con 0x7f62280a0b50 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f621c024ab0 con 0x7f62280a0b50 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 10 client.4167.objecter ms_dispatch 0x7f6228000fd0 osd_map(8..8 src has 1..8) v4 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 3 client.4167.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 3 client.4167.objecter handle_osd_map decoding full epoch 8 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f62257fa700 20 client.4167.objecter dump_active .. 0 homeless 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.553+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622809c330 con 0x7f62280a0b50 2024-09-16T10:49:51.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.557+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f621c043e30 con 0x7f62280a0b50 2024-09-16T10:49:51.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.557+0000 7f62257fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:51.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.557+0000 7f62257fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:51.616 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.610+0000 7fed798612c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T10:49:51.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.621+0000 7f622ec31700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.621+0000 7f622ec31700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.621+0000 7f622ec31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.621+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6228073c20 con 0x7f62280a0b50 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f62257fa700 1 -- 172.21.15.5:0/792347496 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7f621c032030 con 0x7f62280a0b50 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f62257fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:51.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f62257fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:51.632 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:49:51.632 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":8,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:49:51.398636+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:49:51.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 10 librados: watch_flush enter 2024-09-16T10:49:51.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 10 librados: watch_flush exit 2024-09-16T10:49:51.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 20 client.4167.objecter shutdown clearing up homeless session... 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 10 client.4167.objecter successfully canceled tick 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 10 monclient: shutdown 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 msgr2=0x7f6228097940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f621c016000 tx=0x7f621c004770 comp rx=0 tx=0).stop 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 shutdown_connections 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f622813b2f0 0x7f6228091000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62280a0b50 0x7f6228097940 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 --2- 172.21.15.5:0/792347496 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6228098bd0 0x7f6228097400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 >> 172.21.15.5:0/792347496 conn(0x7f62280a1c40 msgr2=0x7f622809c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 shutdown_connections 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 -- 172.21.15.5:0/792347496 wait complete. 2024-09-16T10:49:51.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:51.625+0000 7f622ec31700 1 librados: shutdown 2024-09-16T10:49:51.644 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T10:49:51.644 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T10:49:51.684 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:51.678+0000 7fed798612c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T10:49:51.791 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T10:49:51.786+0000 7f0fbf27a880 -1 Falling back to public interface 2024-09-16T10:49:51.797 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-16T10:49:51.789+0000 7f1f97844880 -1 Falling back to public interface 2024-09-16T10:49:52.047 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T10:49:52.042+0000 7f35cc3a4880 -1 Falling back to public interface 2024-09-16T10:49:52.051 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:49:52.046+0000 7f4853e5a880 -1 Falling back to public interface 2024-09-16T10:49:52.056 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-16T10:49:52.049+0000 7f01945d0880 -1 Falling back to public interface 2024-09-16T10:49:52.080 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-16T10:49:52.073+0000 7fedcad0d880 -1 Falling back to public interface 2024-09-16T10:49:52.114 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:52.110+0000 7fed798612c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T10:49:52.240 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:52.234+0000 7fed798612c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T10:49:52.247 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: get_monmap_and_config 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: build_initial_monmap 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: monmap: 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:52.357951+0000 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:52.357951+0000 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:52.360 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:52.361 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.362 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 1 Processor -- start 2024-09-16T10:49:52.362 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 1 -- start start 2024-09-16T10:49:52.362 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: init 2024-09-16T10:49:52.365 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc9cc0a0890 0x7fc9cc0a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-a con 0x7fc9cc0a0890 addr v2:172.21.15.5:3300/0 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9cc09a0d0 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.353+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9cc09cb90 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc133460 con 0x7fc9cc09cb90 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc09a550 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc09beb0 con 0x7fc9cc0a0890 2024-09-16T10:49:52.366 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient(hunting): authenticate will time out at 850.936121s 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc9cc0a0890 0x7fc9cc0a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1b78700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc9cc09cb90 0x7fc9cc132d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35716/0 (socket says 172.21.15.5:35716) 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1b78700 1 -- 172.21.15.5:0/749496069 learned_addr learned my addr 172.21.15.5:0/749496069 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): get_auth_request con 0x7fc9cc0a0890 auth_method 0 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:52.367 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient(hunting): handle_auth_done global_id 4170 payload 274 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 -- 172.21.15.5:0/749496069 >> v1:172.21.15.143:6789/0 conn(0x7fc9cc09a0d0 legacy=0x7fc9cc132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 -- 172.21.15.5:0/749496069 >> v1:172.21.15.5:6789/0 conn(0x7fc9cc09cb90 legacy=0x7fc9cc132d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient: _finish_hunting 0 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 monclient: found mon.noname-a 2024-09-16T10:49:52.368 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 -- 172.21.15.5:0/749496069 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc09b580 con 0x7fc9cc0a0890 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 1 -- 172.21.15.5:0/749496069 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596313417 0 0) 0x7fc9cc133460 con 0x7fc9cc09cb90 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d1377700 1 --2- 172.21.15.5:0/749496069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc0a0c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619362fb2042b9e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 1 -- 172.21.15.5:0/749496069 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800e070 con 0x7fc9cc0a0890 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:52.369 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: dump: 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:52.370 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: _finish_auth 0 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: _check_auth_tickets 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 5 monclient: authenticate success, global_id 4170 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 1 -- 172.21.15.5:0/749496069 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c800bbd0 con 0x7fc9cc0a0890 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 4 set_mon_vals no callback set 2024-09-16T10:49:52.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 1 -- 172.21.15.5:0/749496069 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8012430 con 0x7fc9cc0a0890 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9c3fff700 10 monclient: dump: 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:52.372 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient: shutdown 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- 172.21.15.5:0/749496069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 msgr2=0x7fc9cc0a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 --2- 172.21.15.5:0/749496069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc0a0c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009d90 tx=0x7fc9c800b330 comp rx=0 tx=0).stop 2024-09-16T10:49:52.373 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- 172.21.15.5:0/749496069 shutdown_connections 2024-09-16T10:49:52.374 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 --2- 172.21.15.5:0/749496069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc0a0c70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.374 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- 172.21.15.5:0/749496069 >> 172.21.15.5:0/749496069 conn(0x7fc9cc0a1fb0 msgr2=0x7fc9cc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:52.374 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- 172.21.15.5:0/749496069 shutdown_connections 2024-09-16T10:49:52.374 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 1 -- 172.21.15.5:0/749496069 wait complete. 2024-09-16T10:49:52.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient: build_initial_monmap 2024-09-16T10:49:52.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.357+0000 7fc9d2379700 10 monclient: monmap: 2024-09-16T10:49:52.379 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-16T10:49:52.379 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 librados: starting msgr at 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 librados: starting objecter 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 Processor -- start 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- start start 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 librados: setting wanted keys 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 librados: calling monclient init 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: init 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-c con 0x7fc9cc09a0d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:52.381 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 0x7fc9cc1cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-b con 0x7fc9cc09cb90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc122060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): picked mon.noname-a con 0x7fc9cc0a0890 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc1257c0 con 0x7fc9cc0a0890 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc09e6a0 con 0x7fc9cc09cb90 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc09c0e0 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:52.382 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient(hunting): authenticate will time out at 850.941254s 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38652/0 (socket says 172.21.15.5:38652) 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 -- 172.21.15.5:0/4157114186 learned_addr learned my addr 172.21.15.5:0/4157114186 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): get_auth_request con 0x7fc9cc09a0d0 auth_method 0 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:52.383 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 0x7fc9cc1cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): get_auth_request con 0x7fc9cc0a0890 auth_method 0 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): get_auth_request con 0x7fc9cc09cb90 auth_method 0 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:52.384 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient(hunting): handle_auth_done global_id 4148 payload 824 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 -- 172.21.15.5:0/4157114186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 msgr2=0x7fc9cc1cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 0x7fc9cc1cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.385 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 -- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 msgr2=0x7fc9cc122060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc122060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient: _finish_hunting 0 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 monclient: found mon.noname-c 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 -- 172.21.15.5:0/4157114186 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c8009a70 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d0b76700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 0x7fc9cc1cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1b78700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc122060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:49:52.386 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d1377700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000f80 tx=0x7fc9c8004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 client.?.objecter ms_handle_connect 0x7fc9cc09a0d0 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c801bb90 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: dump: 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.387 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: _finish_auth 0 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 5 monclient: authenticate success, global_id 4148 2024-09-16T10:49:52.388 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _renew_subs 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc123d00 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c8020070 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c80098f0 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 monclient: dump: 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:52.389 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c37fe700 4 set_mon_vals no callback set 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:52.390 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _renew_subs 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9cc126060 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9d2379700 1 librados: init done 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc9c8026700 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc9c801cb30 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 10 client.4148.objecter ms_dispatch 0x7fc9cc000fd0 osd_map(8..8 src has 1..8) v4 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 3 client.4148.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 3 client.4148.objecter handle_osd_map decoding full epoch 8 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.361+0000 7fc9c1ffb700 20 client.4148.objecter dump_active .. 0 homeless 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.365+0000 7fc9d2379700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.365+0000 7fc9d2379700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:52.391 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.365+0000 7fc9d2379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:52.392 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.365+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9cc125b00 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.392 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.369+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fc9cc125b00 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.392 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.369+0000 7fc9c1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:52.392 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.369+0000 7fc9c1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:52.444 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:52.444 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:52.444 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:52.444 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9cc0a1540 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.445 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9c1ffb700 1 -- 172.21.15.5:0/4157114186 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7fc9c8022070 con 0x7fc9cc09a0d0 2024-09-16T10:49:52.445 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9c1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:52.445 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9c1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:52.445 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-16T10:49:52.445 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":8,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:49:51.398636+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:49:52.447 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 librados: watch_flush enter 2024-09-16T10:49:52.447 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 librados: watch_flush exit 2024-09-16T10:49:52.447 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 20 client.4148.objecter shutdown clearing up homeless session... 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 client.4148.objecter successfully canceled tick 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 10 monclient: shutdown 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 msgr2=0x7fc9cc1cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.437+0000 7fc9d2379700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000f80 tx=0x7fc9c8004af0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 shutdown_connections 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc9cc0a0890 0x7fc9cc122060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9cc09cb90 0x7fc9cc1cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 --2- 172.21.15.5:0/4157114186 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc9cc09a0d0 0x7fc9cc1cab10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 >> 172.21.15.5:0/4157114186 conn(0x7fc9cc0a1fb0 msgr2=0x7fc9cc132e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:52.448 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 shutdown_connections 2024-09-16T10:49:52.449 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 -- 172.21.15.5:0/4157114186 wait complete. 2024-09-16T10:49:52.449 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:52.441+0000 7fc9d2379700 1 librados: shutdown 2024-09-16T10:49:52.461 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T10:49:52.604 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:52.598+0000 7fed798612c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T10:49:52.790 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:52.786+0000 7fed798612c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:49:52.859 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:52.854+0000 7fed798612c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T10:49:53.022 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:53.018+0000 7fed798612c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T10:49:53.757 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:53.750+0000 7fed798612c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T10:49:53.827 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:53.822+0000 7fed798612c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T10:49:53.990 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:53.986+0000 7fed798612c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T10:49:54.057 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:54.054+0000 7fed798612c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T10:49:54.125 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:54.122+0000 7fed798612c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T10:49:54.450 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:54.446+0000 7fed798612c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T10:49:54.529 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:54.522+0000 7fed798612c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T10:49:55.253 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:55.246+0000 7fed798612c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T10:49:55.587 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T10:49:55.582+0000 7f0fbf27a880 -1 osd.3 0 log_to_monitors true 2024-09-16T10:49:55.612 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T10:49:55.606+0000 7f35cc3a4880 -1 osd.5 0 log_to_monitors true 2024-09-16T10:49:55.641 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-16T10:49:55.633+0000 7f1f97844880 -1 osd.1 0 log_to_monitors true 2024-09-16T10:49:55.777 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-16T10:49:55.769+0000 7f01945d0880 -1 osd.2 0 log_to_monitors true 2024-09-16T10:49:55.794 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:49:55.790+0000 7f4853e5a880 -1 osd.4 0 log_to_monitors true 2024-09-16T10:49:55.887 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-16T10:49:55.877+0000 7fedcad0d880 -1 osd.0 0 log_to_monitors true 2024-09-16T10:49:56.298 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:56.294+0000 7fed603d8700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T10:49:56.298 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T10:49:56.294+0000 7fed603d8700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T10:49:57.365 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-16T10:49:57.357+0000 7f1f92e60700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T10:49:57.366 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-16T10:49:57.357+0000 7fedc6329700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T10:49:57.366 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T10:49:57.362+0000 7f0fba896700 -1 osd.3 0 waiting for initial osdmap 2024-09-16T10:49:57.366 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T10:49:57.362+0000 7f35c79c0700 -1 osd.5 0 waiting for initial osdmap 2024-09-16T10:49:57.370 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-16T10:49:57.361+0000 7fedc2606700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:57.372 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-16T10:49:57.365+0000 7f1f8f13d700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:57.376 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T10:49:57.370+0000 7f0fb6b73700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:57.379 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T10:49:57.374+0000 7f35c3c9d700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:58.106 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-16T10:49:58.097+0000 7f0191597700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T10:49:58.113 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-09-16T10:49:58.105+0000 7f0187ec1700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:58.216 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:49:58.210+0000 7f484f476700 -1 osd.4 0 waiting for initial osdmap 2024-09-16T10:49:58.222 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:49:58.218+0000 7f484774b700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:49:59.067 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:49:59.175 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: get_monmap_and_config 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: build_initial_monmap 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: monmap: 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:59.173463+0000 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:59.173463+0000 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:59.176 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:59.177 INFO:teuthology.misc.health.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:59.177 INFO:teuthology.misc.health.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:59.177 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 Processor -- start 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 -- start start 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: init 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4560134d50 0x7f4560135130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-c con 0x7f4560134d50 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f456009ae20 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f456009b230 addr v1:172.21.15.143:6789/0 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560136830 con 0x7f456009ae20 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560099fd0 con 0x7f456009b230 2024-09-16T10:49:59.181 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456009a2b0 con 0x7f4560134d50 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f4565fe8700 10 monclient(hunting): authenticate will time out at 857.750679s 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455effd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f456009ae20 0x7f4560135a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45480/0 (socket says 172.21.15.5:45480) 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455effd700 1 -- 172.21.15.5:0/1946833413 learned_addr learned my addr 172.21.15.5:0/1946833413 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 1 --2- 172.21.15.5:0/1946833413 >> v2:172.21.15.143:3300/0 conn(0x7f4560134d50 0x7f4560135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): get_auth_request con 0x7f4560134d50 auth_method 0 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318497557 0 0) 0x7f4560136830 con 0x7f456009ae20 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 10 monclient(hunting): _negotiate 2024-09-16T10:49:59.182 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 10 monclient(hunting): my global_id is 4212 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45480035d0 con 0x7f456009ae20 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient(hunting): handle_auth_done global_id 4216 payload 274 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 1 -- 172.21.15.5:0/1946833413 >> v1:172.21.15.143:6789/0 conn(0x7f456009b230 legacy=0x7f4560136120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 1 -- 172.21.15.5:0/1946833413 >> v1:172.21.15.5:6789/0 conn(0x7f456009ae20 legacy=0x7f4560135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4285305789 0 0) 0x7f45480035d0 con 0x7f456009ae20 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:49:59.183 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 1 monclient: found mon.noname-c 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455f7fe700 1 -- 172.21.15.5:0/1946833413 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456009a590 con 0x7f4560134d50 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.169+0000 7f455e7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455f7fe700 1 --2- 172.21.15.5:0/1946833413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560134d50 0x7f4560135130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4550009850 tx=0x7f4550009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2207b0e3c5be3277 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554001670 con 0x7f4560134d50 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: dump: 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.184 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 5 monclient: authenticate success, global_id 4216 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4550004030 con 0x7f4560134d50 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:59.185 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 4 set_mon_vals no callback set 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 1 -- 172.21.15.5:0/1946833413 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45500043c0 con 0x7f4560134d50 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f455e7fc700 10 monclient: dump: 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.186 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 10 monclient: shutdown 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- 172.21.15.5:0/1946833413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560134d50 msgr2=0x7f4560135130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 --2- 172.21.15.5:0/1946833413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560134d50 0x7f4560135130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4550009850 tx=0x7f4550009b60 comp rx=0 tx=0).stop 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- 172.21.15.5:0/1946833413 shutdown_connections 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 --2- 172.21.15.5:0/1946833413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560134d50 0x7f4560135130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- 172.21.15.5:0/1946833413 >> 172.21.15.5:0/1946833413 conn(0x7f45600a4ec0 msgr2=0x7f45600a52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- 172.21.15.5:0/1946833413 shutdown_connections 2024-09-16T10:49:59.187 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- 172.21.15.5:0/1946833413 wait complete. 2024-09-16T10:49:59.192 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 10 monclient: build_initial_monmap 2024-09-16T10:49:59.192 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 10 monclient: monmap: 2024-09-16T10:49:59.192 INFO:teuthology.misc.health.smithi005.stderr:epoch 0 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:last_changed 0.000000 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:created 0.000000 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 1 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 librados: starting msgr at 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 librados: starting objecter 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 Processor -- start 2024-09-16T10:49:59.193 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 -- start start 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 librados: setting wanted keys 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 1 librados: calling monclient init 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.173+0000 7f4565fe8700 10 monclient: init 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f456009ae20 0x7f45601caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-c con 0x7f456009ae20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f456009b230 0x7f45601caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-b con 0x7f456009b230 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): picked mon.noname-a con 0x7f4560134d50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600a69c0 con 0x7f4560134d50 2024-09-16T10:49:59.194 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600a2290 con 0x7f456009b230 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456009f3d0 con 0x7f456009ae20 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient(hunting): authenticate will time out at 857.756605s 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f456009ae20 0x7f45601caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36190/0 (socket says 172.21.15.5:36190) 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 -- 172.21.15.5:0/3675303766 learned_addr learned my addr 172.21.15.5:0/3675303766 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): get_auth_request con 0x7f4560134d50 auth_method 0 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.195 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): get_auth_request con 0x7f456009ae20 auth_method 0 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f456009b230 0x7f45601caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): get_auth_request con 0x7f456009b230 auth_method 0 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.196 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 -- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f456009ae20 msgr2=0x7f45601caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f456009ae20 0x7f45601caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 -- 172.21.15.5:0/3675303766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f456009b230 msgr2=0x7f45601caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f456009b230 0x7f45601caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 monclient: found mon.noname-a 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:59.197 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 -- 172.21.15.5:0/3675303766 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45500094b0 con 0x7f4560134d50 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455ffff700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f455800cb40 tx=0x7f455800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 client.?.objecter ms_handle_connect 0x7f4560134d50 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558003c20 con 0x7f4560134d50 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: dump: 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.198 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: _finish_auth 0 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: _check_auth_tickets 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 5 monclient: authenticate success, global_id 4215 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558003d80 con 0x7f4560134d50 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _renew_subs 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:59.199 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45601239d0 con 0x7f4560134d50 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455800aa20 con 0x7f4560134d50 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 monclient: dump: 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:epoch 1 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.200 INFO:teuthology.misc.health.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr: 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _renew_subs 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45601251d0 con 0x7f4560134d50 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4565fe8700 1 librados: init done 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f455d7fa700 4 set_mon_vals no callback set 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) v1 ==== 43216+0+0 (secure 0 0 0) 0x7f455800ab80 con 0x7f4560134d50 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 --2- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 0x7f454c039390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.201 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2767+0+0 (secure 0 0 0) 0x7f4558047200 con 0x7f4560134d50 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 10 client.4215.objecter ms_dispatch 0x7f4560000fd0 osd_map(11..11 src has 1..11) v4 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 3 client.4215.objecter handle_osd_map got epochs [11,11] > 0 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 3 client.4215.objecter handle_osd_map decoding full epoch 11 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.177+0000 7f4547fff700 20 client.4215.objecter dump_active .. 0 homeless 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f455f7fe700 1 --2- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 0x7f454c039390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f455f7fe700 10 monclient: get_auth_request con 0x7f454c036ed0 auth_method 0 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f4565fe8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f4565fe8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f4565fe8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560135130 con 0x7f4560134d50 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f4547fff700 10 client.4215.objecter ms_handle_connect 0x7f454c036ed0 2024-09-16T10:49:59.202 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.181+0000 7f455f7fe700 1 --2- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 0x7f454c039390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45500090b0 tx=0x7f4550008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.203 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.189+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4558012420 con 0x7f4560134d50 2024-09-16T10:49:59.203 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.189+0000 7f4547fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.203 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.189+0000 7f4547fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:59.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.365+0000 7f4565fe8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:59.371 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.365+0000 7f4565fe8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:59.372 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.365+0000 7f4565fe8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:49:59.372 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.365+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4560122e60 con 0x7f4560134d50 2024-09-16T10:49:59.375 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4547fff700 1 -- 172.21.15.5:0/3675303766 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7032 (secure 0 0 0) 0x7f455801a360 con 0x7f4560134d50 2024-09-16T10:49:59.376 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4547fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:49:59.376 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4547fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:49:59.376 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-16T10:49:59.376 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":11,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:49:58.356658+0000","last_up_change":"2024-09-16T10:49:58.356658+0000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2408116882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":2408116882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":2408116882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":2408116882}]},"public_addr":"172.21.15.5:6808/2408116882","cluster_addr":"172.21.15.5:6809/2408116882","heartbeat_back_addr":"172.21.15.5:6811/2408116882","heartbeat_front_addr":"172.21.15.5:6810/2408116882","state":["exists","up"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3675136489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":3675136489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":3675136489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3675136489}]},"public_addr":"172.21.15.5:6800/3675136489","cluster_addr":"172.21.15.5:6801/3675136489","heartbeat_back_addr":"172.21.15.5:6803/3675136489","heartbeat_front_addr":"172.21.15.5:6802/3675136489","state":["exists","up"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2015722671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":2015722671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":2015722671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2015722671}]},"public_addr":"172.21.15.5:6804/2015722671","cluster_addr":"172.21.15.5:6805/2015722671","heartbeat_back_addr":"172.21.15.5:6807/2015722671","heartbeat_front_addr":"172.21.15.5:6806/2015722671","state":["exists","up"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3342511476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6801","nonce":3342511476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6803","nonce":3342511476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3342511476}]},"public_addr":"172.21.15.143:6800/3342511476","cluster_addr":"172.21.15.143:6801/3342511476","heartbeat_back_addr":"172.21.15.143:6803/3342511476","heartbeat_front_addr":"172.21.15.143:6802/3342511476","state":["exists","up"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3348750750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":3348750750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6811","nonce":3348750750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3348750750}]},"public_addr":"172.21.15.143:6808/3348750750","cluster_addr":"172.21.15.143:6809/3348750750","heartbeat_back_addr":"172.21.15.143:6811/3348750750","heartbeat_front_addr":"172.21.15.143:6810/3348750750","state":["exists","up"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3538568827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6805","nonce":3538568827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6807","nonce":3538568827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3538568827}]},"public_addr":"172.21.15.143:6804/3538568827","cluster_addr":"172.21.15.143:6805/3538568827","heartbeat_back_addr":"172.21.15.143:6807/3538568827","heartbeat_front_addr":"172.21.15.143:6806/3538568827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 10 librados: watch_flush enter 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 10 librados: watch_flush exit 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 20 client.4215.objecter shutdown clearing up homeless session... 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 10 client.4215.objecter successfully canceled tick 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 msgr2=0x7f454c039390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 0x7f454c039390 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45500090b0 tx=0x7f4550008030 comp rx=0 tx=0).stop 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 10 monclient: shutdown 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 msgr2=0x7f45601212a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f455800cb40 tx=0x7f455800ce50 comp rx=0 tx=0).stop 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 shutdown_connections 2024-09-16T10:49:59.379 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> v2:172.21.15.143:6812/1098205034 conn(0x7f454c036ed0 0x7f454c039390 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4560134d50 0x7f45601212a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f456009b230 0x7f45601caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 --2- 172.21.15.5:0/3675303766 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f456009ae20 0x7f45601caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 >> 172.21.15.5:0/3675303766 conn(0x7f45600a4ec0 msgr2=0x7f456006b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 shutdown_connections 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 -- 172.21.15.5:0/3675303766 wait complete. 2024-09-16T10:49:59.380 INFO:teuthology.misc.health.smithi005.stderr:2024-09-16T10:49:59.369+0000 7f4565fe8700 1 librados: shutdown 2024-09-16T10:49:59.391 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T10:49:59.391 INFO:tasks.ceph:Creating RBD pool 2024-09-16T10:49:59.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: get_monmap_and_config 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: build_initial_monmap 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: monmap: 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:59.497739+0000 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:59.497739+0000 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:49:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:49:59.501 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:49:59.501 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:49:59.501 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 Processor -- start 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 -- start start 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: init 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d400a38d0 addr v1:172.21.15.5:6789/0 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d400a21c0 0x7f5d400a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-c con 0x7f5d400a21c0 addr v2:172.21.15.143:3300/0 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d4013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-b con 0x7f5d4013aa60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d400a3cb0 con 0x7f5d400a38d0 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d400a35a0 con 0x7f5d400a21c0 2024-09-16T10:49:59.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401362d0 con 0x7f5d4013aa60 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d453b9700 10 monclient(hunting): authenticate will time out at 858.074892s 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d4013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d3ffff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f5d400a38d0 0x7f5d40135bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45482/0 (socket says 172.21.15.5:45482) 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d3ffff700 1 -- 172.21.15.5:0/3479987589 learned_addr learned my addr 172.21.15.5:0/3479987589 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): get_auth_request con 0x7f5d4013aa60 auth_method 0 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 1 --2- 172.21.15.5:0/3479987589 >> v2:172.21.15.143:3300/0 conn(0x7f5d400a21c0 0x7f5d400a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): get_auth_request con 0x7f5d400a21c0 auth_method 0 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 -- 172.21.15.5:0/3479987589 >> v2:172.21.15.143:3300/0 conn(0x7f5d400a21c0 msgr2=0x7f5d400a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d3f7fe700 1 -- 172.21.15.5:0/3479987589 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868912442 0 0) 0x7f5d400a3cb0 con 0x7f5d400a38d0 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 --2- 172.21.15.5:0/3479987589 >> v2:172.21.15.143:3300/0 conn(0x7f5d400a21c0 0x7f5d400a25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 -- 172.21.15.5:0/3479987589 >> v1:172.21.15.5:6789/0 conn(0x7f5d400a38d0 legacy=0x7f5d40135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient: _finish_hunting 0 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 monclient: found mon.noname-b 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d44bb8700 1 -- 172.21.15.5:0/3479987589 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4009f9e0 con 0x7f5d4013aa60 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.493+0000 7f5d3f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d44bb8700 1 --2- 172.21.15.5:0/3479987589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d4013ce50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009d90 tx=0x7f5d3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bef48c705690fb15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 1 -- 172.21.15.5:0/3479987589 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3800e070 con 0x7f5d4013aa60 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: dump: 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:59.509 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: _finish_auth 0 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 1 -- 172.21.15.5:0/3479987589 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3800bb20 con 0x7f5d4013aa60 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 5 monclient: authenticate success, global_id 4190 2024-09-16T10:49:59.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 10 monclient: get_monmap_and_config success 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 4 set_mon_vals no callback set 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 1 -- 172.21.15.5:0/3479987589 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38012430 con 0x7f5d4013aa60 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d3f7fe700 10 monclient: dump: 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.511 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 10 monclient: shutdown 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- 172.21.15.5:0/3479987589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 msgr2=0x7f5d4013ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 --2- 172.21.15.5:0/3479987589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d4013ce50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009d90 tx=0x7f5d3800b330 comp rx=0 tx=0).stop 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- 172.21.15.5:0/3479987589 shutdown_connections 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 --2- 172.21.15.5:0/3479987589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d4013ce50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 --2- 172.21.15.5:0/3479987589 >> v2:172.21.15.143:3300/0 conn(0x7f5d400a21c0 0x7f5d400a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- 172.21.15.5:0/3479987589 >> 172.21.15.5:0/3479987589 conn(0x7f5d400a8440 msgr2=0x7f5d400a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:49:59.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- 172.21.15.5:0/3479987589 shutdown_connections 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- 172.21.15.5:0/3479987589 wait complete. 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 10 monclient: build_initial_monmap 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 10 monclient: monmap: 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:49:59.513 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 librados: starting msgr at 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 librados: starting objecter 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 Processor -- start 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 -- start start 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 librados: setting wanted keys 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 1 librados: calling monclient init 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.497+0000 7f5d453b9700 10 monclient: init 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _reopen_session rank -1 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d400a21c0 0x7f5d401caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-b con 0x7f5d400a21c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d400a38d0 0x7f5d401caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-a con 0x7f5d400a38d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): picked mon.noname-c con 0x7f5d4013aa60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40070fe0 con 0x7f5d400a38d0 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40139ea0 con 0x7f5d400a21c0 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4013a180 con 0x7f5d4013aa60 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): _renew_subs 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient(hunting): authenticate will time out at 858.080591s 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d400a38d0 0x7f5d401caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:34840/0 (socket says 172.21.15.5:34840) 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 -- 172.21.15.5:0/186792500 learned_addr learned my addr 172.21.15.5:0/186792500 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): get_auth_request con 0x7f5d4013aa60 auth_method 0 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d400a21c0 0x7f5d401caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): get_auth_request con 0x7f5d400a38d0 auth_method 0 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): get_auth_request con 0x7f5d400a21c0 auth_method 0 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d37fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 -- 172.21.15.5:0/186792500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d400a21c0 msgr2=0x7f5d401caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d400a21c0 0x7f5d401caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 -- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d400a38d0 msgr2=0x7f5d401caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d400a38d0 0x7f5d401caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient: _finish_hunting 0 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 monclient: found mon.noname-c 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 -- 172.21.15.5:0/186792500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d38009a70 con 0x7f5d4013aa60 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d44bb8700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800b680 tx=0x7f5d38004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 client.?.objecter ms_handle_connect 0x7f5d4013aa60 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38012430 con 0x7f5d4013aa60 2024-09-16T10:49:59.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: dump: 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: _finish_auth 0 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: _check_auth_tickets 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 5 monclient: authenticate success, global_id 4193 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d38020070 con 0x7f5d4013aa60 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _renew_subs 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d401389f0 con 0x7f5d4013aa60 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38004280 con 0x7f5d4013aa60 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:49:59.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 10 monclient: dump: 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3e7fc700 4 set_mon_vals no callback set 2024-09-16T10:49:59.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _renew_subs 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d400a11e0 con 0x7f5d4013aa60 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d453b9700 1 librados: init done 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.501+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 4) v1 ==== 43216+0+0 (secure 0 0 0) 0x7f5d380043e0 con 0x7f5d4013aa60 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d453b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d453b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d453b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 1 --2- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 0x7f5d20039390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d400a25d0 con 0x7f5d4013aa60 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3142+0+0 (secure 0 0 0) 0x7f5d38022070 con 0x7f5d4013aa60 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 10 client.4193.objecter ms_dispatch 0x7f5d40000fd0 osd_map(12..12 src has 1..12) v4 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 3 client.4193.objecter handle_osd_map got epochs [12,12] > 0 2024-09-16T10:49:59.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 3 client.4193.objecter handle_osd_map decoding full epoch 12 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 20 client.4193.objecter dump_active .. 0 homeless 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3ffff700 1 --2- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 0x7f5d20039390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3ffff700 10 monclient: get_auth_request con 0x7f5d20036ed0 auth_method 0 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3d7fa700 10 client.4193.objecter ms_handle_connect 0x7f5d20036ed0 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.505+0000 7f5d3ffff700 1 --2- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 0x7f5d20039390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d30000f60 tx=0x7f5d30007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.513+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5d38004690 con 0x7f5d4013aa60 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.513+0000 7f5d3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:49:59.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.513+0000 7f5d3d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:49:59.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.685+0000 7f5d453b9700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:49:59.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.685+0000 7f5d453b9700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:49:59.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.685+0000 7f5d453b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:49:59.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:49:59.685+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f5d40138770 con 0x7f5d4013aa60 2024-09-16T10:50:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.377+0000 7f5d3d7fa700 1 -- 172.21.15.5:0/186792500 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f5d380257f0 con 0x7f5d4013aa60 2024-09-16T10:50:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.377+0000 7f5d3d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T10:50:00.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.377+0000 7f5d3d7fa700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T10:50:00.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 10 librados: watch_flush enter 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 10 librados: watch_flush exit 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 20 client.4193.objecter shutdown clearing up homeless session... 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 10 client.4193.objecter successfully canceled tick 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 msgr2=0x7f5d20039390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 0x7f5d20039390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d30000f60 tx=0x7f5d30007e00 comp rx=0 tx=0).stop 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 10 monclient: shutdown 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 msgr2=0x7f5d401367e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.381+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800b680 tx=0x7f5d38004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 shutdown_connections 2024-09-16T10:50:00.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d4013aa60 0x7f5d401367e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5d20036ed0 0x7f5d20039390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d400a38d0 0x7f5d401caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 --2- 172.21.15.5:0/186792500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d400a21c0 0x7f5d401caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.385+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 >> 172.21.15.5:0/186792500 conn(0x7f5d400a8440 msgr2=0x7f5d400a7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.389+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 shutdown_connections 2024-09-16T10:50:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.389+0000 7f5d453b9700 1 -- 172.21.15.5:0/186792500 wait complete. 2024-09-16T10:50:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.389+0000 7f5d453b9700 1 librados: shutdown 2024-09-16T10:50:00.397 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2024-09-16T10:50:00.410 DEBUG:teuthology.orchestra.run.smithi005:> rbd --cluster ceph pool init rbd 2024-09-16T10:50:00.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: get_monmap_and_config 2024-09-16T10:50:00.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: build_initial_monmap 2024-09-16T10:50:00.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: monmap: 2024-09-16T10:50:00.456 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:00.451255+0000 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:00.451255+0000 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:00.457 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 Processor -- start 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 -- start start 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: init 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: _reopen_session rank -1 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 --2- >> v2:172.21.15.143:3300/0 conn(0x5574433d04c0 0x5574433d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-c con 0x5574433d04c0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-c-legacy con 0x5574433d0de0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:00.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 0x557443463e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-b con 0x557443463a70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55744324cca0 con 0x5574433d0de0 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x55744328f540 con 0x5574433d04c0 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x55744328cb10 con 0x557443463a70 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): _renew_subs 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.445+0000 7f4409ba3780 10 monclient(hunting): authenticate will time out at 859.027998s 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f44037fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x5574433d0de0 0x557443463360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:50476/0 (socket says 172.21.15.5:50476) 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x5574433d04c0 0x5574433d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f44037fe700 1 -- 172.21.15.5:0/1894180329 learned_addr learned my addr 172.21.15.5:0/1894180329 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 0x557443463e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): get_auth_request con 0x5574433d04c0 auth_method 0 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): get_auth_request con 0x557443463a70 auth_method 0 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:00.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 -- 172.21.15.5:0/1894180329 >> v2:172.21.15.143:3300/0 conn(0x5574433d04c0 msgr2=0x5574433d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 --2- 172.21.15.5:0/1894180329 >> v2:172.21.15.143:3300/0 conn(0x5574433d04c0 0x5574433d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 1 -- 172.21.15.5:0/1894180329 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864466538 0 0) 0x55744324cca0 con 0x5574433d0de0 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 -- 172.21.15.5:0/1894180329 >> v1:172.21.15.143:6789/0 conn(0x5574433d0de0 legacy=0x557443463360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 monclient: found mon.noname-b 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 -- 172.21.15.5:0/1894180329 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557443225bb0 con 0x557443463a70 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f440993c700 1 --2- 172.21.15.5:0/1894180329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 0x557443463e80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43fc005fd0 tx=0x7f43fc00a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d9d1ae0742d139c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:00.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 1 -- 172.21.15.5:0/1894180329 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00f790 con 0x557443463a70 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: dump: 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:00.465 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 5 monclient: authenticate success, global_id 4196 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 1 -- 172.21.15.5:0/1894180329 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc00f8f0 con 0x557443463a70 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 1 -- 172.21.15.5:0/1894180329 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00fc80 con 0x557443463a70 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 10 monclient: get_monmap_and_config success 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 4 set_mon_vals no callback set 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:00.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4402ffd700 10 monclient: dump: 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 10 monclient: shutdown 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:00.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 1 -- 172.21.15.5:0/1894180329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 msgr2=0x557443463e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.449+0000 7f4409ba3780 1 --2- 172.21.15.5:0/1894180329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 0x557443463e80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43fc005fd0 tx=0x7f43fc00a780 comp rx=0 tx=0).stop 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 -- 172.21.15.5:0/1894180329 shutdown_connections 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 --2- 172.21.15.5:0/1894180329 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x557443463a70 0x557443463e80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 --2- 172.21.15.5:0/1894180329 >> v2:172.21.15.143:3300/0 conn(0x5574433d04c0 0x5574433d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 -- 172.21.15.5:0/1894180329 >> 172.21.15.5:0/1894180329 conn(0x5574433b2ed0 msgr2=0x5574433b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 -- 172.21.15.5:0/1894180329 shutdown_connections 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 -- 172.21.15.5:0/1894180329 wait complete. 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 10 monclient: build_initial_monmap 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 10 monclient: monmap: 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:00.468 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 librados: starting msgr at 2024-09-16T10:50:00.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 librados: starting objecter 2024-09-16T10:50:00.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 Processor -- start 2024-09-16T10:50:00.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 -- start start 2024-09-16T10:50:00.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 librados: setting wanted keys 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 1 librados: calling monclient init 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 10 monclient: init 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.453+0000 7f4409ba3780 10 monclient: _reopen_session rank -1 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5574433d04c0 0x557443461ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-c con 0x5574433d04c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574433d0de0 0x55744344b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-b con 0x5574433d0de0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557443463a70 0x55744344b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): picked mon.noname-a con 0x557443463a70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:00.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x557443288400 con 0x557443463a70 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574432449d0 con 0x5574433d0de0 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x557443244670 con 0x5574433d04c0 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): _renew_subs 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient(hunting): authenticate will time out at 859.036275s 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5574433d04c0 0x557443461ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5574433d04c0 0x557443461ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:34856/0 (socket says 172.21.15.5:34856) 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 1 -- 172.21.15.5:0/2365116707 learned_addr learned my addr 172.21.15.5:0/2365116707 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): get_auth_request con 0x5574433d04c0 auth_method 0 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:00.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557443463a70 0x55744344b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574433d0de0 0x55744344b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4403fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): get_auth_request con 0x557443463a70 auth_method 0 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): get_auth_request con 0x5574433d0de0 auth_method 0 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:00.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 -- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5574433d04c0 msgr2=0x557443461ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5574433d04c0 0x557443461ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 -- 172.21.15.5:0/2365116707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574433d0de0 msgr2=0x55744344b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574433d0de0 0x55744344b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:00.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 monclient: found mon.noname-a 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc00a100 con 0x557443463a70 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f440993c700 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557443463a70 0x55744344b640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f43fc012000 tx=0x7f43fc003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 client.?.objecter ms_handle_connect 0x557443463a70 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc01b620 con 0x557443463a70 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: dump: 2024-09-16T10:50:00.477 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: _finish_auth 0 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 5 monclient: authenticate success, global_id 4224 2024-09-16T10:50:00.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _renew_subs 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc01b7a0 con 0x557443463a70 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557443232b00 con 0x557443463a70 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc01bb50 con 0x557443463a70 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f44027fc700 4 set_mon_vals no callback set 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 monclient: dump: 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:00.479 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _renew_subs 2024-09-16T10:50:00.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557443225f40 con 0x557443463a70 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 librados: init done 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 librados: wait_for_osdmap waiting 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) v1 ==== 43216+0+0 (secure 0 0 0) 0x7f43fc01bcb0 con 0x557443463a70 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6812/1098205034 conn(0x7f43e403d750 0x7f43e403fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3495+0+0 (secure 0 0 0) 0x7f43fc047530 con 0x557443463a70 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_map(13..13 src has 1..13) v4 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map got epochs [13,13] > 0 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map decoding full epoch 13 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f43fbfff700 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 librados: wait_for_osdmap done waiting 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:50:00.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.461+0000 7f4403fff700 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6812/1098205034 conn(0x7f43e403d750 0x7f43e403fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.457+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x5574432259a0 con 0x557443463a70 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.461+0000 7f4403fff700 10 monclient: get_auth_request con 0x7f43e403d750 auth_method 0 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.461+0000 7f43fbfff700 10 client.4224.objecter ms_handle_connect 0x7f43e403d750 2024-09-16T10:50:00.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:00.461+0000 7f4403fff700 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6812/1098205034 conn(0x7f43e403d750 0x7f43e403fc10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005fd0 tx=0x7f43f400c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:01.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (secure 0 0 0) 0x7f43fc029070 con 0x557443463a70 2024-09-16T10:50:01.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T10:50:01.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T10:50:01.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f4409ba3780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:01.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55744324bdd0 con 0x557443463a70 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43fc0233b0 con 0x557443463a70 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f44027fc700 10 client.4224.objecter _get_latest_version latest 14, waiting 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f44027fc700 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f44027fc700 10 monclient: _renew_subs 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f44027fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:01.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f44027fc700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=14}) v3 -- 0x7f43e8003c90 con 0x557443463a70 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1028+0+0 (secure 0 0 0) 0x7f43fc023770 con 0x557443463a70 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_map(14..14 src has 1..14) v4 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map got epochs [14,14] > 13 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map decoding incremental epoch 14 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f43fbfff700 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.389+0000 7f4409ba3780 10 librados: create oid=rbd_trash nspace= 2024-09-16T10:50:01.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 10 client.4224.objecter _op_submit op 0x55744344bb80 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 20 client.4224.objecter _calc_target epoch 14 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 20 client.4224.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 10 client.4224.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6804/3538568827 conn(0x5574433cc4c0 0x5574433cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 20 client.4224.objecter _get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 10 client.4224.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 20 client.4224.objecter get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 15 client.4224.objecter _session_op_assign 5 1 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 15 client.4224.objecter _send_op 1 to 2.3 on osd.5 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> v2:172.21.15.143:6804/3538568827 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x5574433ccde0 con 0x5574433cc4c0 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f4409ba3780 5 client.4224.objecter 1 in flight 2024-09-16T10:50:01.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f44037fe700 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6804/3538568827 conn(0x5574433cc4c0 0x5574433cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:01.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f44037fe700 10 monclient: get_auth_request con 0x5574433cc4c0 auth_method 0 2024-09-16T10:50:01.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f44037fe700 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6804/3538568827 conn(0x5574433cc4c0 0x5574433cc8a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:01.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.393+0000 7f43fbfff700 10 client.4224.objecter ms_handle_connect 0x5574433cc4c0 2024-09-16T10:50:01.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 1 -- 172.21.15.5:0/2365116707 <== osd.5 v2:172.21.15.143:6804/3538568827 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f43ec00e620 con 0x5574433cc4c0 2024-09-16T10:50:01.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 7 client.4224.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 15 client.4224.objecter handle_osd_op_reply completed tid 1 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 15 client.4224.objecter _finish_op 1 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 15 client.4224.objecter _session_op_remove 5 1 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f44037fe700 5 client.4224.objecter 0 in flight 2024-09-16T10:50:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 10 librados: Objecter returned from create r=0 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 10 client.4224.objecter _op_submit op 0x5574434c8e30 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 20 client.4224.objecter _calc_target epoch 14 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 20 client.4224.objecter _get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 20 client.4224.objecter get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 15 client.4224.objecter _session_op_assign 5 2 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 15 client.4224.objecter _send_op 2 to 2.2 on osd.5 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.405+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> v2:172.21.15.143:6804/3538568827 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x5574434c93c0 con 0x5574433cc4c0 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 5 client.4224.objecter 1 in flight 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 1 -- 172.21.15.5:0/2365116707 <== osd.5 v2:172.21.15.143:6804/3538568827 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f43ec00e620 con 0x5574433cc4c0 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 7 client.4224.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:50:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter handle_osd_op_reply completed tid 2 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter _finish_op 2 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter _session_op_remove 5 2 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 5 client.4224.objecter 0 in flight 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 10 client.4224.objecter _op_submit op 0x5574434c7fc0 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter _calc_target epoch 14 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter _get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-16T10:50:01.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 15 client.4224.objecter _session_op_assign 5 3 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 15 client.4224.objecter _send_op 3 to 2.2 on osd.5 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 --> v2:172.21.15.143:6804/3538568827 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x5574434c84b0 con 0x5574433cc4c0 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4409ba3780 5 client.4224.objecter 1 in flight 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 1 -- 172.21.15.5:0/2365116707 <== osd.5 v2:172.21.15.143:6804/3538568827 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f43ec00e620 con 0x5574433cc4c0 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 7 client.4224.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter handle_osd_op_reply completed tid 3 2024-09-16T10:50:01.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter _finish_op 3 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 15 client.4224.objecter _session_op_remove 5 3 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f44037fe700 5 client.4224.objecter 0 in flight 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4401ffb700 10 client.4224.objecter allocate_selfmanaged_snap; pool: 2 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4401ffb700 10 client.4224.objecter pool_op_submit 4 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4401ffb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:01.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:01.409+0000 7f4401ffb700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f43dc0030f0 con 0x557443463a70 2024-09-16T10:50:02.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.196+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 9 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f43fc0104c0 con 0x557443463a70 2024-09-16T10:50:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 10 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (secure 0 0 0) 0x7f43fc037980 con 0x557443463a70 2024-09-16T10:50:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 pool_op_reply(tid 4 (0) Success v15) v1 2024-09-16T10:50:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 client.4224.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v15) v1 2024-09-16T10:50:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 client.4224.objecter have request 4 at 0x7f43dc003010 Op: create unmanaged snap 2024-09-16T10:50:02.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 20 client.4224.objecter waiting for client to reach epoch 15 before calling back 2024-09-16T10:50:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 monclient: _renew_subs 2024-09-16T10:50:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=15}) v3 -- 0x7f43e4043af0 con 0x557443463a70 2024-09-16T10:50:02.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.396+0000 7f43fbfff700 10 client.4224.objecter done 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 11 ==== osd_map(15..15 src has 1..15) v4 ==== 626+0+0 (secure 0 0 0) 0x7f43fc025180 con 0x557443463a70 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_map(15..15 src has 1..15) v4 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map got epochs [15,15] > 14 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map decoding incremental epoch 15 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f43fbfff700 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 librados: queue_aio_write 0x5574434c4170 completion 0x55744324acd0 write_seq 1 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 10 client.4224.objecter _op_submit op 0x7f43e80062c0 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 client.4224.objecter _calc_target epoch 15 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:50:02.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 client.4224.objecter _get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 client.4224.objecter get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 15 client.4224.objecter _session_op_assign 5 5 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 15 client.4224.objecter _send_op 5 to 2.2 on osd.5 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 1 -- 172.21.15.5:0/2365116707 --> v2:172.21.15.143:6804/3538568827 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f43e80067b0 con 0x5574433cc4c0 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.400+0000 7f44027fc700 5 client.4224.objecter 1 in flight 2024-09-16T10:50:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 1 -- 172.21.15.5:0/2365116707 <== osd.5 v2:172.21.15.143:6804/3538568827 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f43ec00e620 con 0x5574433cc4c0 2024-09-16T10:50:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 2024-09-16T10:50:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T10:50:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 7 client.4224.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-16T10:50:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 10 client.4224.objecter op 1 rval 0 len 0 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 15 client.4224.objecter handle_osd_op_reply completed tid 5 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 15 client.4224.objecter _finish_op 5 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 15 client.4224.objecter _session_op_remove 5 5 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 5 client.4224.objecter 0 in flight 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f44037fe700 20 librados: complete_aio_write 0x55744324acd0 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f4401ffb700 10 client.4224.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f4401ffb700 10 client.4224.objecter pool_op_submit 6 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f4401ffb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:02.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:02.408+0000 7f4401ffb700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) v4 -- 0x7f43dc004570 con 0x557443463a70 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.412+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 12 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (secure 0 0 0) 0x7f43fc037b50 con 0x557443463a70 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 pool_op_reply(tid 6 (0) Success v16) v1 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v16) v1 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter have request 6 at 0x7f43dc004510 Op: delete unmanaged snap 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 20 client.4224.objecter waiting for client to reach epoch 16 before calling back 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:03.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 monclient: _renew_subs 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f43e4043c80 con 0x557443463a70 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter done 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 1 -- 172.21.15.5:0/2365116707 <== mon.0 v2:172.21.15.5:3300/0 13 ==== osd_map(16..16 src has 1..16) v4 ==== 654+0+0 (secure 0 0 0) 0x7f43fc02d3e0 con 0x557443463a70 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map got epochs [16,16] > 15 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 3 client.4224.objecter handle_osd_map decoding incremental epoch 16 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f43fbfff700 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 librados: queue_aio_write 0x5574434c4170 completion 0x7f43dc001d10 write_seq 2 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 10 client.4224.objecter _op_submit op 0x7f43e80062c0 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 client.4224.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T10:50:03.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 client.4224.objecter _get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 client.4224.objecter get_session s=0x5574433c9df0 osd=5 3 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 15 client.4224.objecter _session_op_assign 5 7 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 15 client.4224.objecter _send_op 7 to 2.2 on osd.5 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 1 -- 172.21.15.5:0/2365116707 --> v2:172.21.15.143:6804/3538568827 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f43e8006b80 con 0x5574433cc4c0 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:03.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.416+0000 7f44027fc700 5 client.4224.objecter 1 in flight 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 1 -- 172.21.15.5:0/2365116707 <== osd.5 v2:172.21.15.143:6804/3538568827 5 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f43ec00e620 con 0x5574433cc4c0 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 10 client.4224.objecter ms_dispatch 0x5574432b8180 osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 7 client.4224.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 15 client.4224.objecter handle_osd_op_reply completed tid 7 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 15 client.4224.objecter _finish_op 7 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 4 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 15 client.4224.objecter _session_op_remove 5 7 2024-09-16T10:50:03.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 5 client.4224.objecter 0 in flight 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f44037fe700 20 librados: complete_aio_write 0x7f43dc001d10 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.420+0000 7f4409ba3780 10 librados: watch_flush enter 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 10 librados: watch_flush exit 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 10 client.4224.objecter close_session for osd.5 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6804/3538568827 conn(0x5574433cc4c0 msgr2=0x5574433cc8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6804/3538568827 conn(0x5574433cc4c0 0x5574433cc8a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 20 client.4224.objecter put_session s=0x5574433c9df0 osd=5 1 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 20 client.4224.objecter shutdown clearing up homeless session... 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 10 client.4224.objecter successfully canceled tick 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6812/1098205034 conn(0x7f43e403d750 msgr2=0x7f43e403fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 --2- 172.21.15.5:0/2365116707 >> v2:172.21.15.143:6812/1098205034 conn(0x7f43e403d750 0x7f43e403fc10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005fd0 tx=0x7f43f400c040 comp rx=0 tx=0).stop 2024-09-16T10:50:03.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 10 monclient: shutdown 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557443463a70 msgr2=0x55744344b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 --2- 172.21.15.5:0/2365116707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x557443463a70 0x55744344b640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f43fc012000 tx=0x7f43fc003330 comp rx=0 tx=0).stop 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f440993c700 1 -- 172.21.15.5:0/2365116707 reap_dead start 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 shutdown_connections 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 >> 172.21.15.5:0/2365116707 conn(0x5574433b2ed0 msgr2=0x557443466460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 shutdown_connections 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 -- 172.21.15.5:0/2365116707 wait complete. 2024-09-16T10:50:03.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.424+0000 7f4409ba3780 1 librados: shutdown 2024-09-16T10:50:03.437 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T10:50:03.437 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T10:50:03.437 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T10:50:03.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T10:50:03.438 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: get_monmap_and_config 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: build_initial_monmap 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: monmap: 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:03.553805+0000 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:03.553805+0000 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:03.556 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:03.557 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 Processor -- start 2024-09-16T10:50:03.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 -- start start 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: init 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0aac0a2c80 0x7f0aac0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-a con 0x7f0aac0a2c80 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0aac0a35a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 0x7f0aac13dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-c con 0x7f0aac136150 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac0a3db0 con 0x7f0aac0a35a0 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0a4b70 con 0x7f0aac0a2c80 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0a0850 con 0x7f0aac136150 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab13b0700 10 monclient(hunting): authenticate will time out at 862.131130s 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0aac0a2c80 0x7f0aac0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0aac0a2c80 0x7f0aac0a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48740/0 (socket says 172.21.15.5:48740) 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 1 -- 172.21.15.5:0/3051657235 learned_addr learned my addr 172.21.15.5:0/3051657235 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): get_auth_request con 0x7f0aac0a2c80 auth_method 0 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 1 --2- 172.21.15.5:0/3051657235 >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 0x7f0aac13dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): get_auth_request con 0x7f0aac136150 auth_method 0 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.548+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 -- 172.21.15.5:0/3051657235 >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 msgr2=0x7f0aac13dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:03.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 --2- 172.21.15.5:0/3051657235 >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 0x7f0aac13dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 -- 172.21.15.5:0/3051657235 >> v1:172.21.15.143:6789/0 conn(0x7f0aac0a35a0 legacy=0x7f0aac135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 monclient: found mon.noname-a 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 1 -- 172.21.15.5:0/3051657235 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39516654 0 0) 0x7f0aac0a3db0 con 0x7f0aac0a35a0 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 -- 172.21.15.5:0/3051657235 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0a4180 con 0x7f0aac0a2c80 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab0baf700 1 --2- 172.21.15.5:0/3051657235 >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 0x7f0aac13dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aabfff700 1 --2- 172.21.15.5:0/3051657235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac0a3060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0a0470 tx=0x7f0a9c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ad67a3de2b498f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 1 -- 172.21.15.5:0/3051657235 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c011830 con 0x7f0aac0a2c80 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: dump: 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:03.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 5 monclient: authenticate success, global_id 4236 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 1 -- 172.21.15.5:0/3051657235 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c011990 con 0x7f0aac0a2c80 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 4 set_mon_vals no callback set 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 1 -- 172.21.15.5:0/3051657235 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c011d20 con 0x7f0aac0a2c80 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0aab7fe700 10 monclient: dump: 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.567 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 10 monclient: shutdown 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- 172.21.15.5:0/3051657235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 msgr2=0x7f0aac0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/3051657235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac0a3060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0a0470 tx=0x7f0a9c009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- 172.21.15.5:0/3051657235 shutdown_connections 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/3051657235 >> v2:172.21.15.143:3300/0 conn(0x7f0aac136150 0x7f0aac13dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/3051657235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac0a3060 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- 172.21.15.5:0/3051657235 >> 172.21.15.5:0/3051657235 conn(0x7f0aac0a4d80 msgr2=0x7f0aac0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- 172.21.15.5:0/3051657235 shutdown_connections 2024-09-16T10:50:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- 172.21.15.5:0/3051657235 wait complete. 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 10 monclient: build_initial_monmap 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 10 monclient: monmap: 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:03.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 librados: starting msgr at 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 librados: starting objecter 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 Processor -- start 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 -- start start 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 librados: setting wanted keys 2024-09-16T10:50:03.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 1 librados: calling monclient init 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.552+0000 7f0ab13b0700 10 monclient: init 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac1ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-a con 0x7f0aac0a2c80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 0x7f0aac1caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-b con 0x7f0aac0a35a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): picked mon.noname-c con 0x7f0aac136150 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac0a7550 con 0x7f0aac0a2c80 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac124600 con 0x7f0aac0a35a0 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac124cc0 con 0x7f0aac136150 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient(hunting): authenticate will time out at 862.137190s 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac1ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37804/0 (socket says 172.21.15.5:37804) 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 -- 172.21.15.5:0/458950255 learned_addr learned my addr 172.21.15.5:0/458950255 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): get_auth_request con 0x7f0aac136150 auth_method 0 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac1ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48752/0 (socket says 172.21.15.5:48752) 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): get_auth_request con 0x7f0aac0a2c80 auth_method 0 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 0x7f0aac1caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aabfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 10 monclient(hunting): get_auth_request con 0x7f0aac0a35a0 auth_method 0 2024-09-16T10:50:03.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 -- 172.21.15.5:0/458950255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 msgr2=0x7f0aac1caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 0x7f0aac1caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 -- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 msgr2=0x7f0aac1ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac1ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient: _finish_hunting 0 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 monclient: found mon.noname-c 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 -- 172.21.15.5:0/458950255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4009a70 con 0x7f0aac136150 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa3fff700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 0x7f0aac1caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:03.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab0baf700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400efc0 tx=0x7f0aa400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 client.?.objecter ms_handle_connect 0x7f0aac136150 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400cb40 con 0x7f0aac136150 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: dump: 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.580 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 5 monclient: authenticate success, global_id 4211 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400f040 con 0x7f0aac136150 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4003a40 con 0x7f0aac136150 2024-09-16T10:50:03.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aa97fa700 10 monclient: dump: 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0aaa7fc700 4 set_mon_vals no callback set 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _renew_subs 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c009570 con 0x7f0aac136150 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _renew_subs 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac126170 con 0x7f0aac136150 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.556+0000 7f0ab13b0700 1 librados: init done 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0aa4014480 con 0x7f0aac136150 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0ab13b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0ab13b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0ab13b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:03.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac09f4e0 con 0x7f0aac136150 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 1 --2- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 0x7f0a8c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f0aa40481b0 con 0x7f0aac136150 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 10 client.4211.objecter ms_dispatch 0x7f0aac000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 3 client.4211.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 3 client.4211.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 20 client.4211.objecter dump_active .. 0 homeless 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aabfff700 1 --2- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 0x7f0a8c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aabfff700 10 monclient: get_auth_request con 0x7f0a8c036f60 auth_method 0 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aa97fa700 10 client.4211.objecter ms_handle_connect 0x7f0a8c036f60 2024-09-16T10:50:03.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.560+0000 7f0aabfff700 1 --2- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 0x7f0a8c039420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c000c00 tx=0x7f0a9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.568+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0aa4012930 con 0x7f0aac136150 2024-09-16T10:50:03.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.568+0000 7f0aa97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.568+0000 7f0aa97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:03.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.780+0000 7f0ab13b0700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:50:03.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.780+0000 7f0ab13b0700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:50:03.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.780+0000 7f0ab13b0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:03.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.780+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0aac09f0d0 con 0x7f0aac136150 2024-09-16T10:50:03.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.784+0000 7f0aa97fa700 1 -- 172.21.15.5:0/458950255 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83061 (secure 0 0 0) 0x7f0aa4018070 con 0x7f0aac136150 2024-09-16T10:50:03.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.784+0000 7f0aa97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T10:50:03.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.784+0000 7f0aa97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:03.796 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 10 librados: watch_flush enter 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 10 librados: watch_flush exit 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 20 client.4211.objecter shutdown clearing up homeless session... 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 10 client.4211.objecter successfully canceled tick 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 msgr2=0x7f0a8c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 0x7f0a8c039420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c000c00 tx=0x7f0a9c008030 comp rx=0 tx=0).stop 2024-09-16T10:50:03.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 10 monclient: shutdown 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 msgr2=0x7f0aac122520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0aa400efc0 tx=0x7f0aa400c3a0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 shutdown_connections 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aac136150 0x7f0aac122520 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aac0a35a0 0x7f0aac1caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a8c036f60 0x7f0a8c039420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 --2- 172.21.15.5:0/458950255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aac0a2c80 0x7f0aac1ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 >> 172.21.15.5:0/458950255 conn(0x7f0aac0a4d80 msgr2=0x7f0aac0a6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 shutdown_connections 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 -- 172.21.15.5:0/458950255 wait complete. 2024-09-16T10:50:03.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.792+0000 7f0ab13b0700 1 librados: shutdown 2024-09-16T10:50:03.814 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1098205034}]},"active_addr":"172.21.15.143:6812/1098205034","active_change":"2024-09-16T10:49:55.274158+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":4243633172}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2637398601}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3754055543}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1190962955}]}]} 2024-09-16T10:50:03.816 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T10:50:03.816 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T10:50:03.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.920+0000 7fd7cef39700 10 monclient: get_monmap_and_config 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.920+0000 7fd7cef39700 10 monclient: build_initial_monmap 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.920+0000 7fd7cef39700 10 monclient: monmap: 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:03.928666+0000 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:03.928666+0000 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:03.931 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:03.932 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 Processor -- start 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- start start 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient: init 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T10:50:03.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-b con 0x7fd7c80a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7c809f780 0x7fd7c80a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-a con 0x7fd7c809f780 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7c81359b0 0x7fd7c8135de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-c con 0x7fd7c81359b0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c80a3a50 con 0x7fd7c809f780 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c80a3c40 con 0x7fd7c81359b0 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80a4300 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient(hunting): authenticate will time out at 862.505396s 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7c809f780 0x7fd7c80a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37816/0 (socket says 172.21.15.5:37816) 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 1 -- 172.21.15.5:0/4253565716 learned_addr learned my addr 172.21.15.5:0/4253565716 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): get_auth_request con 0x7fd7c80a2bf0 auth_method 0 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): get_auth_request con 0x7fd7c809f780 auth_method 0 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 1 --2- 172.21.15.5:0/4253565716 >> v2:172.21.15.143:3300/0 conn(0x7fd7c81359b0 0x7fd7c8135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): get_auth_request con 0x7fd7c81359b0 auth_method 0 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 -- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 msgr2=0x7fd7c80a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 -- 172.21.15.5:0/4253565716 >> v2:172.21.15.143:3300/0 conn(0x7fd7c81359b0 msgr2=0x7fd7c8135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 --2- 172.21.15.5:0/4253565716 >> v2:172.21.15.143:3300/0 conn(0x7fd7c81359b0 0x7fd7c8135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient: _finish_hunting 0 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 monclient: found mon.noname-a 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 -- 172.21.15.5:0/4253565716 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c809fda0 con 0x7fd7c809f780 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cdf37700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cd736700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809f780 0x7fd7c80a3510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7c809fbb0 tx=0x7fd7b8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf59c18a3dfea301 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 1 -- 172.21.15.5:0/4253565716 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8011830 con 0x7fd7c809f780 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: dump: 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.940 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.941 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: _finish_auth 0 2024-09-16T10:50:03.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: _check_auth_tickets 2024-09-16T10:50:03.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 5 monclient: authenticate success, global_id 4242 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 1 -- 172.21.15.5:0/4253565716 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b8011990 con 0x7fd7c809f780 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 4 set_mon_vals no callback set 2024-09-16T10:50:03.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 1 -- 172.21.15.5:0/4253565716 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8011d20 con 0x7fd7c809f780 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7ccf35700 10 monclient: dump: 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 10 monclient: shutdown 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809f780 msgr2=0x7fd7c80a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809f780 0x7fd7c80a3510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7c809fbb0 tx=0x7fd7b8009b30 comp rx=0 tx=0).stop 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- 172.21.15.5:0/4253565716 shutdown_connections 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- 172.21.15.5:0/4253565716 >> v2:172.21.15.143:3300/0 conn(0x7fd7c81359b0 0x7fd7c8135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c809f780 0x7fd7c80a3510 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 --2- 172.21.15.5:0/4253565716 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c80a2bf0 0x7fd7c80a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- 172.21.15.5:0/4253565716 >> 172.21.15.5:0/4253565716 conn(0x7fd7c80a53e0 msgr2=0x7fd7c8073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- 172.21.15.5:0/4253565716 shutdown_connections 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.924+0000 7fd7cef39700 1 -- 172.21.15.5:0/4253565716 wait complete. 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: build_initial_monmap 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: monmap: 2024-09-16T10:50:03.946 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:03.947 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 librados: starting msgr at 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 librados: starting objecter 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 Processor -- start 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- start start 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 librados: setting wanted keys 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 librados: calling monclient init 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: init 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:03.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7c809f780 0x7fd7c81ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-b con 0x7fd7c809f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-a con 0x7fd7c80a2bf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 0x7fd7c8135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): picked mon.noname-c con 0x7fd7c81359b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c809f4e0 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8120e60 con 0x7fd7c809f780 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c80a67f0 con 0x7fd7c81359b0 2024-09-16T10:50:03.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient(hunting): authenticate will time out at 862.509906s 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 0x7fd7c8135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 0x7fd7c8135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37826/0 (socket says 172.21.15.5:37826) 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48764/0 (socket says 172.21.15.5:48764) 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 1 -- 172.21.15.5:0/3011411293 learned_addr learned my addr 172.21.15.5:0/3011411293 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): get_auth_request con 0x7fd7c81359b0 auth_method 0 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): get_auth_request con 0x7fd7c80a2bf0 auth_method 0 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cdf37700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7c809f780 0x7fd7c81ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cdf37700 10 monclient(hunting): get_auth_request con 0x7fd7c809f780 auth_method 0 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cdf37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cdf37700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cdf37700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7ce738700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:03.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 -- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 msgr2=0x7fd7c8135340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 0x7fd7c8135340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 -- 172.21.15.5:0/3011411293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7c809f780 msgr2=0x7fd7c81ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7c809f780 0x7fd7c81ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient: _finish_hunting 0 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 monclient: found mon.noname-a 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 -- 172.21.15.5:0/3011411293 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0009b30 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cd736700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8074ce0 tx=0x7fd7b8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 client.?.objecter ms_handle_connect 0x7fd7c80a2bf0 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8004510 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: dump: 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.957 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 5 monclient: authenticate success, global_id 4245 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b801e070 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _renew_subs 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b80094b0 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:03.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8003900 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7be7fc700 10 monclient: dump: 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:03.959 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _renew_subs 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c81cab00 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7cef39700 1 librados: init done 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.928+0000 7fd7bffff700 4 set_mon_vals no callback set 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cef39700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cef39700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cef39700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c80704d0 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd7b8003a60 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 1 --2- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 0x7fd7ac039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:03.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7fd7b80483b0 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 10 client.4245.objecter ms_dispatch 0x7fd7c8000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 3 client.4245.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 3 client.4245.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 20 client.4245.objecter dump_active .. 0 homeless 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cdf37700 1 --2- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 0x7fd7ac039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cdf37700 10 monclient: get_auth_request con 0x7fd7ac036f60 auth_method 0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7be7fc700 10 client.4245.objecter ms_handle_connect 0x7fd7ac036f60 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.932+0000 7fd7cdf37700 1 --2- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 0x7fd7ac039420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009320 tx=0x7fd7c000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.940+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd7b802d450 con 0x7fd7c80a2bf0 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.940+0000 7fd7be7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:03.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:03.940+0000 7fd7be7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:04.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7cef39700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7cef39700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7cef39700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7c80a2fd0 con 0x7fd7c80a2bf0 2024-09-16T10:50:04.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7be7fc700 1 -- 172.21.15.5:0/3011411293 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10822 (secure 0 0 0) 0x7fd7b8015070 con 0x7fd7c80a2bf0 2024-09-16T10:50:04.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7be7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.112+0000 7fd7be7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:04.122 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:04.122 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":16,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:50:03.409895+0000","last_up_change":"2024-09-16T10:49:58.356658+0000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:49:59.343692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:49:59.729840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2408116882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":2408116882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":2408116882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":2408116882}]},"public_addr":"172.21.15.5:6808/2408116882","cluster_addr":"172.21.15.5:6809/2408116882","heartbeat_back_addr":"172.21.15.5:6811/2408116882","heartbeat_front_addr":"172.21.15.5:6810/2408116882","state":["exists","up"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3675136489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":3675136489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":3675136489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3675136489}]},"public_addr":"172.21.15.5:6800/3675136489","cluster_addr":"172.21.15.5:6801/3675136489","heartbeat_back_addr":"172.21.15.5:6803/3675136489","heartbeat_front_addr":"172.21.15.5:6802/3675136489","state":["exists","up"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2015722671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":2015722671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":2015722671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2015722671}]},"public_addr":"172.21.15.5:6804/2015722671","cluster_addr":"172.21.15.5:6805/2015722671","heartbeat_back_addr":"172.21.15.5:6807/2015722671","heartbeat_front_addr":"172.21.15.5:6806/2015722671","state":["exists","up"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3342511476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6801","nonce":3342511476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6803","nonce":3342511476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3342511476}]},"public_addr":"172.21.15.143:6800/3342511476","cluster_addr":"172.21.15.143:6801/3342511476","heartbeat_back_addr":"172.21.15.143:6803/3342511476","heartbeat_front_addr":"172.21.15.143:6802/3342511476","state":["exists","up"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3348750750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":3348750750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6811","nonce":3348750750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3348750750}]},"public_addr":"172.21.15.143:6808/3348750750","cluster_addr":"172.21.15.143:6809/3348750750","heartbeat_back_addr":"172.21.15.143:6811/3348750750","heartbeat_front_addr":"172.21.15.143:6810/3348750750","state":["exists","up"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3538568827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6805","nonce":3538568827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6807","nonce":3538568827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3538568827}]},"public_addr":"172.21.15.143:6804/3538568827","cluster_addr":"172.21.15.143:6805/3538568827","heartbeat_back_addr":"172.21.15.143:6807/3538568827","heartbeat_front_addr":"172.21.15.143:6806/3538568827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.912822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.644997+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.809738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.543973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.760941+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.643053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:50:04.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 10 librados: watch_flush enter 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 10 librados: watch_flush exit 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 20 client.4245.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 10 client.4245.objecter successfully canceled tick 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 msgr2=0x7fd7ac039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 0x7fd7ac039420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009320 tx=0x7fd7c000b040 comp rx=0 tx=0).stop 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 10 monclient: shutdown 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 msgr2=0x7fd7c8134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8074ce0 tx=0x7fd7b8004770 comp rx=0 tx=0).stop 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 shutdown_connections 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7c81359b0 0x7fd7c8135340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7ac036f60 0x7fd7ac039420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7c80a2bf0 0x7fd7c8134e00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 --2- 172.21.15.5:0/3011411293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7c809f780 0x7fd7c81ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 >> 172.21.15.5:0/3011411293 conn(0x7fd7c80a53e0 msgr2=0x7fd7c80a6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 shutdown_connections 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 -- 172.21.15.5:0/3011411293 wait complete. 2024-09-16T10:50:04.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.116+0000 7fd7cef39700 1 librados: shutdown 2024-09-16T10:50:04.142 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T10:50:04.142 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:50:04.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: monmap: 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.249390+0000 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.249390+0000 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.252 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.253 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 Processor -- start 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 -- start start 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: init 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T10:50:04.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f197406afe0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1974131e20 0x7f197409e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-a con 0x7f1974131e20 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-b con 0x7f1974137d90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197409ff60 con 0x7f197406afe0 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197409fbb0 con 0x7f1974131e20 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197409bc10 con 0x7f1974137d90 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient(hunting): authenticate will time out at 862.825777s 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1974131e20 0x7f197409e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1974131e20 0x7f197409e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48772/0 (socket says 172.21.15.5:48772) 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37832/0 (socket says 172.21.15.5:37832) 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 -- 172.21.15.5:0/3875018786 learned_addr learned my addr 172.21.15.5:0/3875018786 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1978b44700 1 --1- 172.21.15.5:0/3875018786 >> v1:172.21.15.143:6789/0 conn(0x7f197406afe0 0x7f197409d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:51406/0 (socket says 172.21.15.5:51406) 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): get_auth_request con 0x7f1974131e20 auth_method 0 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): get_auth_request con 0x7f1974137d90 auth_method 0 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 -- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 msgr2=0x7f197413a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 --2- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 -- 172.21.15.5:0/3875018786 >> v1:172.21.15.143:6789/0 conn(0x7f197406afe0 legacy=0x7f197409d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 1 -- 172.21.15.5:0/3875018786 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249091582 0 0) 0x7f197409ff60 con 0x7f197406afe0 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 -- 172.21.15.5:0/3875018786 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197409bef0 con 0x7f1974131e20 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f1973fff700 1 --2- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f197409e0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f195c009910 tx=0x7f195c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e555f56b477d763 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 1 -- 172.21.15.5:0/3875018786 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c011830 con 0x7f1974131e20 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: dump: 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.260 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 5 monclient: authenticate success, global_id 4248 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 1 -- 172.21.15.5:0/3875018786 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c011990 con 0x7f1974131e20 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 1 -- 172.21.15.5:0/3875018786 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c011d40 con 0x7f1974131e20 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f197ada8700 4 set_mon_vals no callback set 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.244+0000 7f19737fe700 10 monclient: dump: 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.262 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 10 monclient: shutdown 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 msgr2=0x7f197409e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 --2- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f197409e0f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f195c009910 tx=0x7f195c009c20 comp rx=0 tx=0).stop 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- 172.21.15.5:0/3875018786 shutdown_connections 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 --2- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 --2- 172.21.15.5:0/3875018786 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f197409e0f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- 172.21.15.5:0/3875018786 >> 172.21.15.5:0/3875018786 conn(0x7f19740a0b30 msgr2=0x7f19740a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- 172.21.15.5:0/3875018786 shutdown_connections 2024-09-16T10:50:04.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- 172.21.15.5:0/3875018786 wait complete. 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 10 monclient: monmap: 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.270 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 librados: starting msgr at 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 librados: starting objecter 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 Processor -- start 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 -- start start 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 librados: setting wanted keys 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 1 librados: calling monclient init 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 10 monclient: init 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.248+0000 7f197ada8700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197406afe0 0x7f1974130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-b con 0x7f197406afe0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-a con 0x7f1974131e20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): picked mon.noname-c con 0x7f1974137d90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197406e840 con 0x7f1974131e20 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974131300 con 0x7f197406afe0 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197406d850 con 0x7f1974137d90 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient(hunting): authenticate will time out at 862.832155s 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48782/0 (socket says 172.21.15.5:48782) 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 1 -- 172.21.15.5:0/2663035741 learned_addr learned my addr 172.21.15.5:0/2663035741 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): get_auth_request con 0x7f1974131e20 auth_method 0 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197406afe0 0x7f1974130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): get_auth_request con 0x7f1974137d90 auth_method 0 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): get_auth_request con 0x7f197406afe0 auth_method 0 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1978b44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 -- 172.21.15.5:0/2663035741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197406afe0 msgr2=0x7f1974130dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197406afe0 0x7f1974130dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 -- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 msgr2=0x7f1974070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 monclient: found mon.noname-c 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 -- 172.21.15.5:0/2663035741 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c009570 con 0x7f1974137d90 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1973fff700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1979345700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f196400e830 tx=0x7f196400eb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 client.?.objecter ms_handle_connect 0x7f1974137d90 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964013680 con 0x7f1974137d90 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: dump: 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.276 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 5 monclient: authenticate success, global_id 4223 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19640137e0 con 0x7f1974137d90 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964013b90 con 0x7f1974137d90 2024-09-16T10:50:04.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _renew_subs 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19740a1f30 con 0x7f1974137d90 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f1972ffd700 4 set_mon_vals no callback set 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 10 monclient: dump: 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.278 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _renew_subs 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197406ed00 con 0x7f1974137d90 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 librados: init done 2024-09-16T10:50:04.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197409d9e0 con 0x7f1974137d90 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.252+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1964013cf0 con 0x7f1974137d90 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 1 --2- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 0x7f1954039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f1964047720 con 0x7f1974137d90 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 10 client.4223.objecter ms_dispatch 0x7f1974000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 3 client.4223.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 3 client.4223.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 20 client.4223.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f1978b44700 1 --2- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 0x7f1954039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f1978b44700 10 monclient: get_auth_request con 0x7f1954036f60 auth_method 0 2024-09-16T10:50:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f1978b44700 1 --2- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 0x7f1954039420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f196c0012e0 tx=0x7f196c001b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.256+0000 7f19717fa700 10 client.4223.objecter ms_handle_connect 0x7f1954036f60 2024-09-16T10:50:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.264+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f196401bbe0 con 0x7f1974137d90 2024-09-16T10:50:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.264+0000 7f19717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.264+0000 7f19717fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f197ada8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f197ada8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f197ada8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f197406d5d0 con 0x7f1974137d90 2024-09-16T10:50:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f19717fa700 1 -- 172.21.15.5:0/2663035741 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10822 (secure 0 0 0) 0x7f1964017090 con 0x7f1974137d90 2024-09-16T10:50:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f19717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.432+0000 7f19717fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:04.442 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:04.442 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":16,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:50:03.409895+0000","last_up_change":"2024-09-16T10:49:58.356658+0000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:49:59.343692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:49:59.729840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2408116882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":2408116882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":2408116882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":2408116882}]},"public_addr":"172.21.15.5:6808/2408116882","cluster_addr":"172.21.15.5:6809/2408116882","heartbeat_back_addr":"172.21.15.5:6811/2408116882","heartbeat_front_addr":"172.21.15.5:6810/2408116882","state":["exists","up"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3675136489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":3675136489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":3675136489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3675136489}]},"public_addr":"172.21.15.5:6800/3675136489","cluster_addr":"172.21.15.5:6801/3675136489","heartbeat_back_addr":"172.21.15.5:6803/3675136489","heartbeat_front_addr":"172.21.15.5:6802/3675136489","state":["exists","up"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2015722671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":2015722671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":2015722671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2015722671}]},"public_addr":"172.21.15.5:6804/2015722671","cluster_addr":"172.21.15.5:6805/2015722671","heartbeat_back_addr":"172.21.15.5:6807/2015722671","heartbeat_front_addr":"172.21.15.5:6806/2015722671","state":["exists","up"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3342511476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6801","nonce":3342511476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6803","nonce":3342511476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3342511476}]},"public_addr":"172.21.15.143:6800/3342511476","cluster_addr":"172.21.15.143:6801/3342511476","heartbeat_back_addr":"172.21.15.143:6803/3342511476","heartbeat_front_addr":"172.21.15.143:6802/3342511476","state":["exists","up"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3348750750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":3348750750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6811","nonce":3348750750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3348750750}]},"public_addr":"172.21.15.143:6808/3348750750","cluster_addr":"172.21.15.143:6809/3348750750","heartbeat_back_addr":"172.21.15.143:6811/3348750750","heartbeat_front_addr":"172.21.15.143:6810/3348750750","state":["exists","up"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3538568827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6805","nonce":3538568827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6807","nonce":3538568827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3538568827}]},"public_addr":"172.21.15.143:6804/3538568827","cluster_addr":"172.21.15.143:6805/3538568827","heartbeat_back_addr":"172.21.15.143:6807/3538568827","heartbeat_front_addr":"172.21.15.143:6806/3538568827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.912822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.644997+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.809738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.543973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.760941+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.643053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:50:04.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 10 librados: watch_flush enter 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 10 librados: watch_flush exit 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 20 client.4223.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 10 client.4223.objecter successfully canceled tick 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 msgr2=0x7f1954039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 0x7f1954039420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f196c0012e0 tx=0x7f196c001b70 comp rx=0 tx=0).stop 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 10 monclient: shutdown 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 msgr2=0x7f197413a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f196400e830 tx=0x7f196400eb40 comp rx=0 tx=0).stop 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 shutdown_connections 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1974137d90 0x7f197413a6c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1954036f60 0x7f1954039420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1974131e20 0x7f1974070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 --2- 172.21.15.5:0/2663035741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197406afe0 0x7f1974130dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 >> 172.21.15.5:0/2663035741 conn(0x7f19740a0b30 msgr2=0x7f19740a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 shutdown_connections 2024-09-16T10:50:04.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 -- 172.21.15.5:0/2663035741 wait complete. 2024-09-16T10:50:04.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.436+0000 7f197ada8700 1 librados: shutdown 2024-09-16T10:50:04.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:50:04.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:50:04.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:50:04.459 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T10:50:04.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T10:50:04.460 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T10:50:04.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.552+0000 7f8275831700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.552+0000 7f8275831700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.552+0000 7f8275831700 10 monclient: monmap: 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.560412+0000 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.560412+0000 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 Processor -- start 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: monmap: 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.561025+0000 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.561025+0000 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.565 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.566 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.566 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.566 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.566 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.566 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 -- start start 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient: init 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8270135d50 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 0x7f827013aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): picked mon.noname-b con 0x7f82700a0520 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f827013afb0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8270071ac0 con 0x7f827013afb0 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8270135750 con 0x7f8270135d50 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82700a0100 con 0x7f82700a0520 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient(hunting): authenticate will time out at 863.137137s 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 Processor -- start 2024-09-16T10:50:04.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275030700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f827013afb0 0x7f827013d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58944/0 (socket says 172.21.15.5:58944) 2024-09-16T10:50:04.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275030700 1 -- 172.21.15.5:0/1601807039 learned_addr learned my addr 172.21.15.5:0/1601807039 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 -- start start 2024-09-16T10:50:04.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: init 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 --2- 172.21.15.5:0/1601807039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 0x7f827013aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): get_auth_request con 0x7f82700a0520 auth_method 0 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 1 -- 172.21.15.5:0/1601807039 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96259777 0 0) 0x7f8270071ac0 con 0x7f827013afb0 2024-09-16T10:50:04.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient(hunting): _negotiate 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient(hunting): my global_id is 4254 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 1 -- 172.21.15.5:0/1601807039 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82580035d0 con 0x7f827013afb0 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 -- 172.21.15.5:0/1601807039 >> v1:172.21.15.143:6789/0 conn(0x7f8270135d50 legacy=0x7f8270136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 -- 172.21.15.5:0/1601807039 >> v1:172.21.15.5:6789/0 conn(0x7f827013afb0 legacy=0x7f827013d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 monclient: found mon.noname-b 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 -- 172.21.15.5:0/1601807039 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270071810 con 0x7f82700a0520 2024-09-16T10:50:04.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826ffff700 1 --2- 172.21.15.5:0/1601807039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 0x7f827013aa70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8260009ae0 tx=0x7f8260009df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19fe6e573603081c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b2009e8e0 0x7f6b2009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2009e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b2009d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b2009dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b268ba700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6b2009dfd0 0x7f6b20133dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58948/0 (socket says 172.21.15.5:58948) 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b268ba700 1 -- 172.21.15.5:0/252306663 learned_addr learned my addr 172.21.15.5:0/252306663 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b20070e40 con 0x7f6b2009dfd0 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b2009e8e0 0x7f6b2009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b201344e0 con 0x7f6b2009d6a0 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200a01d0 con 0x7f6b2009e8e0 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b270bb700 10 monclient(hunting): authenticate will time out at 863.138885s 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 1 -- 172.21.15.5:0/1601807039 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000b850 con 0x7f82700a0520 2024-09-16T10:50:04.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: dump: 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.574 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 1 -- 172.21.15.5:0/1601807039 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826000b9b0 con 0x7f82700a0520 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 5 monclient: authenticate success, global_id 4226 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 1 -- 172.21.15.5:0/1601807039 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000bd40 con 0x7f82700a0520 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 4 set_mon_vals no callback set 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f826f7fe700 10 monclient: dump: 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.576 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699006859 0 0) 0x7f6b20070e40 con 0x7f6b2009dfd0 2024-09-16T10:50:04.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): _negotiate 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): my global_id is 4257 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b0c0035d0 con 0x7f6b2009dfd0 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342937192 0 0) 0x7f6b0c0035d0 con 0x7f6b2009dfd0 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b2009e8e0 msgr2=0x7f6b2009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b2009e8e0 0x7f6b2009ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 >> v1:172.21.15.143:6789/0 conn(0x7f6b2009d6a0 legacy=0x7f6b201336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2009fef0 con 0x7f6b2009dfd0 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f6b180019f0 con 0x7f6b2009dfd0 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: dump: 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.579 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 >> v1:172.21.15.5:6789/0 conn(0x7f6b2009dfd0 legacy=0x7f6b20133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b0c003a50 0x7f6b0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): picked mon.a con 0x7f6b0c003a50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c004870 0x7f6b0c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): picked mon.c con 0x7f6b0c004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c005430 0x7f6b0c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): picked mon.b con 0x7f6b0c005430 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200a01d0 con 0x7f6b0c003a50 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c00c2c0 con 0x7f6b0c005430 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c00c480 con 0x7f6b0c004870 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b250b7700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b260b9700 10 monclient(hunting): get_auth_request con 0x7f6b2009e8e0 auth_method 0 2024-09-16T10:50:04.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b2009e8e0 0x7f6b2009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b0c003a50 0x7f6b0c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): get_auth_request con 0x7f6b0c003a50 auth_method 0 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c004870 0x7f6b0c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): get_auth_request con 0x7f6b0c004870 auth_method 0 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:04.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 10 monclient: shutdown 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 -- 172.21.15.5:0/1601807039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 msgr2=0x7f827013aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.556+0000 7f8275831700 1 --2- 172.21.15.5:0/1601807039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 0x7f827013aa70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8260009ae0 tx=0x7f8260009df0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- 172.21.15.5:0/1601807039 shutdown_connections 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 --2- 172.21.15.5:0/1601807039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f82700a0520 0x7f827013aa70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- 172.21.15.5:0/1601807039 >> 172.21.15.5:0/1601807039 conn(0x7f82700a45d0 msgr2=0x7f82700a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c005430 0x7f6b0c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): get_auth_request con 0x7f6b0c005430 auth_method 0 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:04.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- 172.21.15.5:0/1601807039 shutdown_connections 2024-09-16T10:50:04.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- 172.21.15.5:0/1601807039 wait complete. 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient: monmap: 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 librados: starting msgr at 2024-09-16T10:50:04.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 librados: starting objecter 2024-09-16T10:50:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient(hunting): handle_auth_done global_id 4257 payload 293 2024-09-16T10:50:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 -- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c004870 msgr2=0x7f6b0c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c004870 0x7f6b0c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 Processor -- start 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 -- 172.21.15.5:0/252306663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c005430 msgr2=0x7f6b0c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c005430 0x7f6b0c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 monclient: found mon.a 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 -- 172.21.15.5:0/252306663 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0035d0 con 0x7f6b0c003a50 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 5 monclient: authenticate success, global_id 4257 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b258b8700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c004870 0x7f6b0c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b260b9700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b0c003a50 0x7f6b0c003e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b200a1180 tx=0x7f6b0c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771f2032dc7b2a2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c01eab0 con 0x7f6b0c003a50 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: dump: 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.589 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c01ec10 con 0x7f6b0c003a50 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 4 set_mon_vals no callback set 2024-09-16T10:50:04.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 1 -- 172.21.15.5:0/252306663 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004030 con 0x7f6b0c003a50 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b250b7700 10 monclient: dump: 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.591 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 10 monclient: shutdown 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b0c003a50 msgr2=0x7f6b0c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 --2- 172.21.15.5:0/252306663 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b0c003a50 0x7f6b0c003e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6b200a1180 tx=0x7f6b0c0058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- start start 2024-09-16T10:50:04.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 librados: setting wanted keys 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 librados: calling monclient init 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient: init 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 0x7f8270133f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): picked mon.noname-a con 0x7f82700a0520 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8270135d50 0x7f8270134470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b268ba700 1 -- 172.21.15.5:0/252306663 reap_dead start 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 shutdown_connections 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 >> 172.21.15.5:0/252306663 conn(0x7f6b200a45d0 msgr2=0x7f6b200a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 shutdown_connections 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 -- 172.21.15.5:0/252306663 wait complete. 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 10 monclient: monmap: 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.594 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.595 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.595 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.595 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.595 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): picked mon.noname-c con 0x7f8270135d50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827013afb0 0x7f827011a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): picked mon.noname-b con 0x7f827013afb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270134d60 con 0x7f82700a0520 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827011df70 con 0x7f827013afb0 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827011c640 con 0x7f8270135d50 2024-09-16T10:50:04.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f8275831700 10 monclient(hunting): authenticate will time out at 863.142807s 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 0x7f8270133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 0x7f8270133f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48802/0 (socket says 172.21.15.5:48802) 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 -- 172.21.15.5:0/3239758052 learned_addr learned my addr 172.21.15.5:0/3239758052 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): get_auth_request con 0x7f82700a0520 auth_method 0 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 1 --2- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8270135d50 0x7f8270134470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): get_auth_request con 0x7f8270135d50 auth_method 0 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f826ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 -- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8270135d50 msgr2=0x7f8270134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 --2- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8270135d50 0x7f8270134470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 -- 172.21.15.5:0/3239758052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827013afb0 msgr2=0x7f827011a530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 --2- 172.21.15.5:0/3239758052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827013afb0 0x7f827011a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 -- 172.21.15.5:0/3239758052 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600094b0 con 0x7f82700a0520 2024-09-16T10:50:04.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 librados: starting msgr at 2024-09-16T10:50:04.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f6b270bb700 1 librados: starting objecter 2024-09-16T10:50:04.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.560+0000 7f827482f700 1 --2- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 0x7f8270133f30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8264001b50 tx=0x7f8264001e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 Processor -- start 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- start start 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 librados: setting wanted keys 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 librados: calling monclient init 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: init 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c003a50 0x7f6b201c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-c con 0x7f6b0c003a50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b2009b0d0 0x7f6b201329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2009b0d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 0x7f6b2011c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): picked mon.noname-a con 0x7f6b20132f30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b201c6370 con 0x7f6b20132f30 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b201333e0 con 0x7f6b2009b0d0 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20120420 con 0x7f6b0c003a50 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient(hunting): authenticate will time out at 863.144761s 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 0x7f6b2011c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c003a50 0x7f6b201c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 0x7f6b2011c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48816/0 (socket says 172.21.15.5:48816) 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 -- 172.21.15.5:0/3361240292 learned_addr learned my addr 172.21.15.5:0/3361240292 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): get_auth_request con 0x7f6b20132f30 auth_method 0 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): get_auth_request con 0x7f6b0c003a50 auth_method 0 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b258b8700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b2009b0d0 0x7f6b201329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 -- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c003a50 msgr2=0x7f6b201c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c003a50 0x7f6b201c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 -- 172.21.15.5:0/3361240292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b2009b0d0 msgr2=0x7f6b201329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b2009b0d0 0x7f6b201329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 -- 172.21.15.5:0/3361240292 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2009b7d0 con 0x7f6b20132f30 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b258b8700 10 monclient: get_auth_request con 0x7f6b2009b0d0 auth_method 0 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6b0c003a50 0x7f6b201c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b258b8700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b2009b0d0 0x7f6b201329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b268ba700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 0x7f6b2011c450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b2006b9c0 tx=0x7f6b18001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 client.?.objecter ms_handle_connect 0x7f6b20132f30 2024-09-16T10:50:04.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800a950 con 0x7f6b20132f30 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: dump: 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.608 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 5 monclient: authenticate success, global_id 4263 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _renew_subs 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2011fb10 con 0x7f6b20132f30 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _renew_subs 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2011e050 con 0x7f6b20132f30 2024-09-16T10:50:04.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b270bb700 1 librados: init done 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 client.?.objecter ms_handle_connect 0x7f82700a0520 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82640052d0 con 0x7f82700a0520 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: dump: 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.610 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 5 monclient: authenticate success, global_id 4260 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264005450 con 0x7f82700a0520 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826e7fc700 4 set_mon_vals no callback set 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800aab0 con 0x7f6b20132f30 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800ae60 con 0x7f6b20132f30 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 monclient: dump: 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b17fff700 4 set_mon_vals no callback set 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6b1801d460 con 0x7f6b20132f30 2024-09-16T10:50:04.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6b04036fb0 0x7f6b04039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264006ac0 con 0x7f82700a0520 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 10 monclient: _renew_subs 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827011dc90 con 0x7f82700a0520 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 10 monclient: _renew_subs 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827013e450 con 0x7f82700a0520 2024-09-16T10:50:04.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f8275831700 1 librados: init done 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f6b18047a50 con 0x7f6b20132f30 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 10 client.4263.objecter ms_dispatch 0x7f6b20000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 3 client.4263.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 3 client.4263.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b16ffd700 20 client.4263.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f6b260b9700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6b04036fb0 0x7f6b04039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.564+0000 7f826d7fa700 10 monclient: dump: 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.616 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f6b260b9700 10 monclient: get_auth_request con 0x7f6b04036fb0 auth_method 0 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 10 client.4260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 15 client.4260.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 20 client.4260.objecter _get_session osd=-1 returning homeless 2024-09-16T10:50:04.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f827009f860 con 0x7f82700a0520 2024-09-16T10:50:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f6b16ffd700 10 client.4263.objecter ms_handle_connect 0x7f6b04036fb0 2024-09-16T10:50:04.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f6b260b9700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6b04036fb0 0x7f6b04039470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c008010 tx=0x7f6b1c015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f826400f070 con 0x7f82700a0520 2024-09-16T10:50:04.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6812/1098205034 conn(0x7f825c036fe0 0x7f825c0394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f82640577c0 con 0x7f82700a0520 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 client.4260.objecter ms_dispatch 0x7f8270000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 3 client.4260.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 3 client.4260.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 client.4260.objecter checking command 1 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6808/3348750750 conn(0x7f825c03c920 0x7f825c03ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter _get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 15 client.4260.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter _get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 15 client.4260.objecter _session_command_op_assign 4 1 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 4 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 client.4260.objecter _send_command 1 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 --> v2:172.21.15.143:6808/3348750750 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f825c03f3f0 con 0x7f825c03c920 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 20 client.4260.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8264059dd0 con 0x7f82700a0520 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826ffff700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6812/1098205034 conn(0x7f825c036fe0 0x7f825c0394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826ffff700 10 monclient: get_auth_request con 0x7f825c036fe0 auth_method 0 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275030700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6808/3348750750 conn(0x7f825c03c920 0x7f825c03ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275030700 10 monclient: get_auth_request con 0x7f825c03c920 auth_method 0 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 client.4260.objecter ms_handle_connect 0x7f825c03c920 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f8275030700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6808/3348750750 conn(0x7f825c03c920 0x7f825c03ed60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826d7fa700 10 client.4260.objecter ms_handle_connect 0x7f825c036fe0 2024-09-16T10:50:04.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.568+0000 7f826ffff700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6812/1098205034 conn(0x7f825c036fe0 0x7f825c0394a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f82600090b0 tx=0x7f8260008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 10 client.4263.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6b08002f60 0x7f6b08005420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 20 client.4263.objecter _get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 20 client.4263.objecter _get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 20 client.4263.objecter get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_assign 0 1 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 4 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 10 client.4263.objecter _send_command 1 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 --> v2:172.21.15.5:6808/2408116882 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b08008480 con 0x7f6b08002f60 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b258b8700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6b08002f60 0x7f6b08005420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b258b8700 10 monclient: get_auth_request con 0x7f6b08002f60 auth_method 0 2024-09-16T10:50:04.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== osd.4 v2:172.21.15.143:6808/3348750750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f825c03f3f0 con 0x7f825c03c920 2024-09-16T10:50:04.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f826d7fa700 10 client.4260.objecter ms_dispatch 0x7f8270000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f826d7fa700 10 client.4260.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f826d7fa700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 4 2024-09-16T10:50:04.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f826d7fa700 15 client.4260.objecter _session_command_op_remove 4 1 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b16ffd700 10 client.4263.objecter ms_handle_connect 0x7f6b08002f60 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.572+0000 7f6b258b8700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6b08002f60 0x7f6b08005420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.576+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== osd.0 v2:172.21.15.5:6808/2408116882 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f6b08008480 con 0x7f6b08002f60 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.576+0000 7f6b16ffd700 10 client.4263.objecter ms_dispatch 0x7f6b20000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.576+0000 7f6b16ffd700 10 client.4263.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.576+0000 7f6b16ffd700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 4 2024-09-16T10:50:04.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.576+0000 7f6b16ffd700 15 client.4263.objecter _session_command_op_remove 0 1 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 10 client.4260.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 15 client.4260.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 20 client.4260.objecter _get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 20 client.4260.objecter _get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 15 client.4260.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 20 client.4260.objecter get_session s=0x7f8264005450 osd=4 3 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 15 client.4260.objecter _session_command_op_assign 4 2 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 4 2024-09-16T10:50:04.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 10 client.4260.objecter _send_command 2 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 --> v2:172.21.15.143:6808/3348750750 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f827011e0b0 con 0x7f825c03c920 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f826d7fa700 1 -- 172.21.15.5:0/3239758052 <== osd.4 v2:172.21.15.143:6808/3348750750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f827011e0b0 con 0x7f825c03c920 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f826d7fa700 10 client.4260.objecter ms_dispatch 0x7f8270000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f826d7fa700 10 client.4260.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f826d7fa700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 4 2024-09-16T10:50:04.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.592+0000 7f826d7fa700 15 client.4260.objecter _session_command_op_remove 4 2 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 10 librados: watch_flush enter 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 10 librados: watch_flush exit 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 10 client.4260.objecter close_session for osd.4 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6808/3348750750 conn(0x7f825c03c920 msgr2=0x7f825c03ed60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6808/3348750750 conn(0x7f825c03c920 0x7f825c03ed60 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 20 client.4260.objecter put_session s=0x7f8264005450 osd=4 1 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 20 client.4260.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 10 client.4260.objecter successfully canceled tick 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6812/1098205034 conn(0x7f825c036fe0 msgr2=0x7f825c0394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 --2- 172.21.15.5:0/3239758052 >> v2:172.21.15.143:6812/1098205034 conn(0x7f825c036fe0 0x7f825c0394a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f82600090b0 tx=0x7f8260008030 comp rx=0 tx=0).stop 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 10 monclient: shutdown 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 msgr2=0x7f8270133f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 --2- 172.21.15.5:0/3239758052 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82700a0520 0x7f8270133f30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8264001b50 tx=0x7f8264001e60 comp rx=0 tx=0).stop 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275030700 1 -- 172.21.15.5:0/3239758052 reap_dead start 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 shutdown_connections 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 >> 172.21.15.5:0/3239758052 conn(0x7f82700a45d0 msgr2=0x7f827009b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 shutdown_connections 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 10 client.4263.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 20 client.4263.objecter _get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 20 client.4263.objecter _get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 20 client.4263.objecter get_session s=0x7f6b08002870 osd=0 3 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 15 client.4263.objecter _session_command_op_assign 0 2 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 4 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 10 client.4263.objecter _send_command 2 2024-09-16T10:50:04.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 --> v2:172.21.15.5:6808/2408116882 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b080011a0 con 0x7f6b08002f60 2024-09-16T10:50:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 -- 172.21.15.5:0/3239758052 wait complete. 2024-09-16T10:50:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f8275831700 1 librados: shutdown 2024-09-16T10:50:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f4036e05700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f4036e05700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f4036e05700 10 monclient: monmap: 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.601817+0000 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.601817+0000 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.632 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b16ffd700 1 -- 172.21.15.5:0/3361240292 <== osd.0 v2:172.21.15.5:6808/2408116882 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b080011a0 con 0x7f6b08002f60 2024-09-16T10:50:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b16ffd700 10 client.4263.objecter ms_dispatch 0x7f6b20000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b16ffd700 10 client.4263.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b16ffd700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 4 2024-09-16T10:50:04.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.596+0000 7f6b16ffd700 15 client.4263.objecter _session_command_op_remove 0 2 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 Processor -- start 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- start start 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient: init 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40300a0900 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f40300a13f0 0x7f403013aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-a con 0x7f40300a13f0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f403013b0e0 0x7f403013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-c con 0x7f403013b0e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403009b0e0 con 0x7f40300a0900 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40300a17d0 con 0x7f40300a13f0 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403009a720 con 0x7f403013b0e0 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient(hunting): authenticate will time out at 863.180712s 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f40300a13f0 0x7f403013aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f40300a13f0 0x7f403013aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48826/0 (socket says 172.21.15.5:48826) 2024-09-16T10:50:04.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 -- 172.21.15.5:0/3294116719 learned_addr learned my addr 172.21.15.5:0/3294116719 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): get_auth_request con 0x7f40300a13f0 auth_method 0 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 1 --2- 172.21.15.5:0/3294116719 >> v2:172.21.15.143:3300/0 conn(0x7f403013b0e0 0x7f403013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): get_auth_request con 0x7f403013b0e0 auth_method 0 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 -- 172.21.15.5:0/3294116719 >> v2:172.21.15.143:3300/0 conn(0x7f403013b0e0 msgr2=0x7f403013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 --2- 172.21.15.5:0/3294116719 >> v2:172.21.15.143:3300/0 conn(0x7f403013b0e0 0x7f403013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 -- 172.21.15.5:0/3294116719 >> v1:172.21.15.143:6789/0 conn(0x7f40300a0900 legacy=0x7f40300a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 -- 172.21.15.5:0/3294116719 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403009a440 con 0x7f40300a13f0 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: monmap: 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.605889+0000 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.605889+0000 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.639 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.640 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.640 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4035602700 1 --2- 172.21.15.5:0/3294116719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40300a13f0 0x7f403013aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4030073c20 tx=0x7f4020009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce31293f42548d33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 10 librados: watch_flush enter 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 10 librados: watch_flush exit 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 10 client.4263.objecter close_session for osd.0 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6b08002f60 msgr2=0x7f6b08005420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6b08002f60 0x7f6b08005420 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 20 client.4263.objecter put_session s=0x7f6b08002870 osd=0 1 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 20 client.4263.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 10 client.4263.objecter successfully canceled tick 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6b04036fb0 msgr2=0x7f6b04039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 --2- 172.21.15.5:0/3361240292 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6b04036fb0 0x7f6b04039470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c008010 tx=0x7f6b1c015000 comp rx=0 tx=0).stop 2024-09-16T10:50:04.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 10 monclient: shutdown 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 msgr2=0x7f6b2011c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 --2- 172.21.15.5:0/3361240292 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b20132f30 0x7f6b2011c450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b2006b9c0 tx=0x7f6b18001a40 comp rx=0 tx=0).stop 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 Processor -- start 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b268ba700 1 -- 172.21.15.5:0/3361240292 reap_dead start 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 shutdown_connections 2024-09-16T10:50:04.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 >> 172.21.15.5:0/3361240292 conn(0x7f6b200a45d0 msgr2=0x7f6b2009c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 -- start start 2024-09-16T10:50:04.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: init 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 1 -- 172.21.15.5:0/3294116719 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020011830 con 0x7f40300a13f0 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: dump: 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.644 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 5 monclient: authenticate success, global_id 4266 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 1 -- 172.21.15.5:0/3294116719 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4020011990 con 0x7f40300a13f0 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 1 -- 172.21.15.5:0/3294116719 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020011830 con 0x7f40300a13f0 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4034e01700 10 monclient: dump: 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 4 set_mon_vals no callback set 2024-09-16T10:50:04.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 shutdown_connections 2024-09-16T10:50:04.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 -- 172.21.15.5:0/3361240292 wait complete. 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 0x7f1ddc136010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-b con 0x7f1ddc135c30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ddc0a04f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ddc070e60 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc135750 con 0x7f1ddc0a04f0 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc071b30 con 0x7f1ddc070e60 2024-09-16T10:50:04.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc09f830 con 0x7f1ddc135c30 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de3194700 10 monclient(hunting): authenticate will time out at 863.182986s 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de1991700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1ddc0a04f0 0x7f1ddc070750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58964/0 (socket says 172.21.15.5:58964) 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 0x7f1ddc136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de1991700 1 -- 172.21.15.5:0/1802889041 learned_addr learned my addr 172.21.15.5:0/1802889041 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): get_auth_request con 0x7f1ddc135c30 auth_method 0 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 10 monclient: shutdown 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- 172.21.15.5:0/3294116719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40300a13f0 msgr2=0x7f403013aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 --2- 172.21.15.5:0/3294116719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40300a13f0 0x7f403013aba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4030073c20 tx=0x7f4020009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f6b270bb700 1 librados: shutdown 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 -- 172.21.15.5:0/1802889041 >> v1:172.21.15.143:6789/0 conn(0x7f1ddc070e60 legacy=0x7f1ddc13da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 -- 172.21.15.5:0/1802889041 >> v1:172.21.15.5:6789/0 conn(0x7f1ddc0a04f0 legacy=0x7f1ddc070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 monclient: found mon.noname-b 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 -- 172.21.15.5:0/1802889041 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc09f1c0 con 0x7f1ddc135c30 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- 172.21.15.5:0/3294116719 shutdown_connections 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 --2- 172.21.15.5:0/3294116719 >> v2:172.21.15.143:3300/0 conn(0x7f403013b0e0 0x7f403013d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 --2- 172.21.15.5:0/3294116719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f40300a13f0 0x7f403013aba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f4036e05700 1 -- 172.21.15.5:0/3294116719 >> 172.21.15.5:0/3294116719 conn(0x7f403006a4f0 msgr2=0x7f403006a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f1de2192700 1 --2- 172.21.15.5:0/1802889041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 0x7f1ddc136010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009dc0 tx=0x7f1dd400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f75fef7d996fa63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- 172.21.15.5:0/3294116719 shutdown_connections 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 1 -- 172.21.15.5:0/1802889041 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400e070 con 0x7f1ddc135c30 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: dump: 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.653 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 1 -- 172.21.15.5:0/1802889041 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd400bad0 con 0x7f1ddc135c30 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 1 -- 172.21.15.5:0/1802889041 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400be60 con 0x7f1ddc135c30 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de1190700 10 monclient: dump: 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 5 monclient: authenticate success, global_id 4235 2024-09-16T10:50:04.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 4 set_mon_vals no callback set 2024-09-16T10:50:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- 172.21.15.5:0/3294116719 wait complete. 2024-09-16T10:50:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f3e52e45700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f3e52e45700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.600+0000 7f3e52e45700 10 monclient: monmap: 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.608786+0000 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.608786+0000 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.657 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 Processor -- start 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 -- start start 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient: init 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: monmap: 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 10 monclient: shutdown 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- 172.21.15.5:0/1802889041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 msgr2=0x7f1ddc136010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 --2- 172.21.15.5:0/1802889041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 0x7f1ddc136010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009dc0 tx=0x7f1dd400b330 comp rx=0 tx=0).stop 2024-09-16T10:50:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 librados: starting msgr at 2024-09-16T10:50:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 librados: starting objecter 2024-09-16T10:50:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 Processor -- start 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- start start 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 librados: setting wanted keys 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 librados: calling monclient init 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: init 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e4c09e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3e4c09f3d0 0x7f3e4c13f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c09f3d0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e4c13f6a0 0x7f3e4c141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-a con 0x7f3e4c13f6a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c09bf80 con 0x7f3e4c09e8e0 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c09b930 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c09bc10 con 0x7f3e4c09f3d0 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient(hunting): authenticate will time out at 863.185525s 2024-09-16T10:50:04.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51e43700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3e4c09e8e0 0x7f3e4c09ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58980/0 (socket says 172.21.15.5:58980) 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51e43700 1 -- 172.21.15.5:0/2903658268 learned_addr learned my addr 172.21.15.5:0/2903658268 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 --2- 172.21.15.5:0/2903658268 >> v2:172.21.15.5:3300/0 conn(0x7f3e4c13f6a0 0x7f3e4c141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): get_auth_request con 0x7f3e4c13f6a0 auth_method 0 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 1 --2- 172.21.15.5:0/2903658268 >> v2:172.21.15.143:3300/0 conn(0x7f3e4c09f3d0 0x7f3e4c13f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): get_auth_request con 0x7f3e4c09f3d0 auth_method 0 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- 172.21.15.5:0/1802889041 shutdown_connections 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 --2- 172.21.15.5:0/1802889041 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc135c30 0x7f1ddc136010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- 172.21.15.5:0/1802889041 >> 172.21.15.5:0/1802889041 conn(0x7f1ddc0a45d0 msgr2=0x7f1ddc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- 172.21.15.5:0/1802889041 shutdown_connections 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 1 -- 172.21.15.5:0/2903658268 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017158749 0 0) 0x7f3e4c09bf80 con 0x7f3e4c09e8e0 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient(hunting): _negotiate 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient(hunting): my global_id is 4272 2024-09-16T10:50:04.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 1 -- 172.21.15.5:0/2903658268 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e380035d0 con 0x7f3e4c09e8e0 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40300a0900 0x7f4030136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-c con 0x7f40300a0900 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40300a13f0 0x7f4030123d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-b con 0x7f40300a13f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 0x7f4030124290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): picked mon.noname-a con 0x7f403013b0e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403009cd40 con 0x7f403013b0e0 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403009c0e0 con 0x7f40300a13f0 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403009c7a0 con 0x7f40300a0900 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient(hunting): authenticate will time out at 863.186236s 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 0x7f4030124290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 0x7f4030124290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48832/0 (socket says 172.21.15.5:48832) 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 -- 172.21.15.5:0/3261605638 learned_addr learned my addr 172.21.15.5:0/3261605638 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): get_auth_request con 0x7f403013b0e0 auth_method 0 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 -- 172.21.15.5:0/2903658268 >> v2:172.21.15.143:3300/0 conn(0x7f3e4c09f3d0 msgr2=0x7f3e4c13f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 --2- 172.21.15.5:0/2903658268 >> v2:172.21.15.143:3300/0 conn(0x7f3e4c09f3d0 0x7f3e4c13f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 -- 172.21.15.5:0/2903658268 >> v1:172.21.15.5:6789/0 conn(0x7f3e4c09e8e0 legacy=0x7f3e4c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 -- 172.21.15.5:0/2903658268 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c09b3e0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- 172.21.15.5:0/1802889041 wait complete. 2024-09-16T10:50:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52644700 1 --2- 172.21.15.5:0/2903658268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c141a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009850 tx=0x7f3e3c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c2785f99d0f635 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 -- 172.21.15.5:0/3261605638 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40300a0900 msgr2=0x7f4030136380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 --2- 172.21.15.5:0/3261605638 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f40300a0900 0x7f4030136380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 -- 172.21.15.5:0/3261605638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40300a13f0 msgr2=0x7f4030123d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 --2- 172.21.15.5:0/3261605638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40300a13f0 0x7f4030123d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 -- 172.21.15.5:0/3261605638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020009570 con 0x7f403013b0e0 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 10 monclient: monmap: 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.671 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 1 -- 172.21.15.5:0/2903658268 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c011830 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: dump: 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.672 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 1 -- 172.21.15.5:0/2903658268 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c011990 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 1 -- 172.21.15.5:0/2903658268 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c011d00 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 5 monclient: authenticate success, global_id 4275 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e52e45700 4 set_mon_vals no callback set 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f3e50e41700 10 monclient: dump: 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 librados: starting msgr at 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 librados: starting objecter 2024-09-16T10:50:04.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 Processor -- start 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036604700 1 --2- 172.21.15.5:0/3261605638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 0x7f4030124290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f402800efc0 tx=0x7f402800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 client.?.objecter ms_handle_connect 0x7f403013b0e0 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402800cd40 con 0x7f403013b0e0 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: dump: 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.676 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402800f040 con 0x7f403013b0e0 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 5 monclient: authenticate success, global_id 4278 2024-09-16T10:50:04.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _renew_subs 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4030126a90 con 0x7f403013b0e0 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280146c0 con 0x7f403013b0e0 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f40267fc700 10 monclient: dump: 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.678 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4027fff700 4 set_mon_vals no callback set 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _renew_subs 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403013a690 con 0x7f403013b0e0 2024-09-16T10:50:04.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f4036e05700 1 librados: init done 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: shutdown 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- 172.21.15.5:0/2903658268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 msgr2=0x7f3e4c141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2903658268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c141a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009850 tx=0x7f3e3c009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- 172.21.15.5:0/2903658268 shutdown_connections 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2903658268 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c141a90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2903658268 >> v2:172.21.15.143:3300/0 conn(0x7f3e4c09f3d0 0x7f3e4c13f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- 172.21.15.5:0/2903658268 >> 172.21.15.5:0/2903658268 conn(0x7f3e4c0a45d0 msgr2=0x7f3e4c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 10 client.4278.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 15 client.4278.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 20 client.4278.objecter _get_session osd=-1 returning homeless 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f403009c3a0 con 0x7f403013b0e0 2024-09-16T10:50:04.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 -- start start 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 librados: setting wanted keys 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 1 librados: calling monclient init 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.604+0000 7f1de3194700 10 monclient: init 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ddc070e60 0x7f1ddc1346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-a con 0x7f1ddc070e60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 0x7f1ddc134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-c con 0x7f1ddc0a04f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ddc135c30 0x7f1ddc11e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): picked mon.noname-b con 0x7f1ddc135c30 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc1206d0 con 0x7f1ddc070e60 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc09b620 con 0x7f1ddc135c30 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc09b0d0 con 0x7f1ddc0a04f0 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient(hunting): authenticate will time out at 863.188492s 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 0x7f1ddc134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ddc070e60 0x7f1ddc1346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 0x7f1ddc134c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37898/0 (socket says 172.21.15.5:37898) 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 1 -- 172.21.15.5:0/896493289 learned_addr learned my addr 172.21.15.5:0/896493289 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): get_auth_request con 0x7f1ddc0a04f0 auth_method 0 2024-09-16T10:50:04.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): get_auth_request con 0x7f1ddc070e60 auth_method 0 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- 172.21.15.5:0/2903658268 shutdown_connections 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 -- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 msgr2=0x7f1ddc134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 --2- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 0x7f1ddc134c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 -- 172.21.15.5:0/896493289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ddc135c30 msgr2=0x7f1ddc11e1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 --2- 172.21.15.5:0/896493289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ddc135c30 0x7f1ddc11e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 -- 172.21.15.5:0/896493289 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4009a70 con 0x7f1ddc070e60 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 1 --2- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ddc0a04f0 0x7f1ddc134c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- 172.21.15.5:0/2903658268 wait complete. 2024-09-16T10:50:04.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4028014820 con 0x7f403013b0e0 2024-09-16T10:50:04.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de2192700 1 --2- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ddc070e60 0x7f1ddc1346c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4014000 tx=0x7f1dd4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 client.?.objecter ms_handle_connect 0x7f1ddc070e60 2024-09-16T10:50:04.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4014036f60 0x7f4014039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f40280484e0 con 0x7f403013b0e0 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 10 client.4278.objecter ms_dispatch 0x7f4030000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 3 client.4278.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 3 client.4278.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40048a0 con 0x7f1ddc070e60 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: dump: 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.689 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4012450 con 0x7f1ddc070e60 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 5 monclient: authenticate success, global_id 4281 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _renew_subs 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc1220a0 con 0x7f1ddc070e60 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40127e0 con 0x7f1ddc070e60 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 monclient: dump: 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _renew_subs 2024-09-16T10:50:04.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc09d050 con 0x7f1ddc070e60 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd3fff700 4 set_mon_vals no callback set 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de3194700 1 librados: init done 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: monmap: 2024-09-16T10:50:04.692 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 librados: starting msgr at 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 librados: starting objecter 2024-09-16T10:50:04.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 Processor -- start 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 10 client.4278.objecter checking command 1 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.5:6800/3675136489 conn(0x7f401403ca10 0x7f401403ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter _get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 15 client.4278.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter _get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 15 client.4278.objecter _session_command_op_assign 1 1 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 4 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 10 client.4278.objecter _send_command 1 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 --> v2:172.21.15.5:6800/3675136489 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f401403f4e0 con 0x7f401403ca10 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 20 client.4278.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f402804aa80 con 0x7f403013b0e0 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1dd4019030 con 0x7f1ddc070e60 2024-09-16T10:50:04.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- start start 2024-09-16T10:50:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 librados: setting wanted keys 2024-09-16T10:50:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 librados: calling monclient init 2024-09-16T10:50:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: init 2024-09-16T10:50:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4035602700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.5:6800/3675136489 conn(0x7f401403ca10 0x7f401403ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4035602700 10 monclient: get_auth_request con 0x7f401403ca10 auth_method 0 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1dc0036f60 0x7f1dc0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f1dd404c220 con 0x7f1ddc070e60 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 10 client.4281.objecter ms_dispatch 0x7f1ddc000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 3 client.4281.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 3 client.4281.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1dd2ffd700 20 client.4281.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e4c09e8e0 0x7f3e4c1399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-b con 0x7f3e4c09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e4c09f3d0 0x7f3e4c139f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-c con 0x7f3e4c09f3d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c122d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): picked mon.noname-a con 0x7f3e4c13f6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0a2aa0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c124f70 con 0x7f3e4c09e8e0 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c126490 con 0x7f3e4c09f3d0 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52e45700 10 monclient(hunting): authenticate will time out at 863.191673s 2024-09-16T10:50:04.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f4035602700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.5:6800/3675136489 conn(0x7f401403ca10 0x7f401403ee50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f40267fc700 10 client.4278.objecter ms_handle_connect 0x7f401403ca10 2024-09-16T10:50:04.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1dc0036f60 0x7f1dc0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f1de1991700 10 monclient: get_auth_request con 0x7f1dc0036f60 auth_method 0 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e4c09f3d0 0x7f3e4c139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e4c09f3d0 0x7f3e4c139f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37906/0 (socket says 172.21.15.5:37906) 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 1 -- 172.21.15.5:0/2647981288 learned_addr learned my addr 172.21.15.5:0/2647981288 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 10 monclient(hunting): get_auth_request con 0x7f3e4c09f3d0 auth_method 0 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e51642700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f3e52644700 1 --2- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c122d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): get_auth_request con 0x7f3e4c13f6a0 auth_method 0 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51642700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f1de1991700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1dc0036f60 0x7f1dc0039420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007d00 tx=0x7f1dd8007c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f1dd2ffd700 10 client.4281.objecter ms_handle_connect 0x7f1dc0036f60 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 -- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e4c09f3d0 msgr2=0x7f3e4c139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 --2- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e4c09f3d0 0x7f3e4c139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 -- 172.21.15.5:0/2647981288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e4c09e8e0 msgr2=0x7f3e4c1399c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T10:50:04.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 --2- 172.21.15.5:0/2647981288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e4c09e8e0 0x7f3e4c1399c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 -- 172.21.15.5:0/2647981288 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0094b0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.705 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:04.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T10:50:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== osd.1 v2:172.21.15.5:6800/3675136489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f401403f4e0 con 0x7f401403ca10 2024-09-16T10:50:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f40267fc700 10 client.4278.objecter ms_dispatch 0x7f4030000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f40267fc700 10 client.4278.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f40267fc700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 4 2024-09-16T10:50:04.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f40267fc700 15 client.4278.objecter _session_command_op_remove 1 1 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52644700 1 --2- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c122d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c016000 tx=0x7f3e3c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 client.?.objecter ms_handle_connect 0x7f3e4c13f6a0 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c006ea0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: dump: 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.707 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 5 monclient: authenticate success, global_id 4284 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 10 monclient: _renew_subs 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c126770 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0045d0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c003900 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e43fff700 4 set_mon_vals no callback set 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 monclient: dump: 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.709 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 10 monclient: _renew_subs 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c124cb0 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e52e45700 1 librados: init done 2024-09-16T10:50:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f5079e3c700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f5079e3c700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.608+0000 7f5079e3c700 10 monclient: monmap: 2024-09-16T10:50:04.711 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.616774+0000 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.616774+0000 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 Processor -- start 2024-09-16T10:50:04.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 -- start start 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient: init 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f507408a410 0x7f507408a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-a con 0x7f507408a410 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f507409e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f507409f400 0x7f507414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-c con 0x7f507409f400 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507409fb80 con 0x7f507409e8e0 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507409c030 con 0x7f507408a410 2024-09-16T10:50:04.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507409b970 con 0x7f507409f400 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient(hunting): authenticate will time out at 863.193572s 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f507408a410 0x7f507408a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5073fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f507409e8e0 0x7f507409ecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:51448/0 (socket says 172.21.15.5:51448) 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5073fff700 1 -- 172.21.15.5:0/824558422 learned_addr learned my addr 172.21.15.5:0/824558422 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): get_auth_request con 0x7f507408a410 auth_method 0 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3e3c021070 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-09-16T10:50:04.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 -- 172.21.15.5:0/824558422 >> v2:172.21.15.143:3300/0 conn(0x7f507409f400 msgr2=0x7f507414a5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 --2- 172.21.15.5:0/824558422 >> v2:172.21.15.143:3300/0 conn(0x7f507409f400 0x7f507414a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 -- 172.21.15.5:0/824558422 >> v1:172.21.15.143:6789/0 conn(0x7f507409e8e0 legacy=0x7f507409ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 -- 172.21.15.5:0/824558422 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507409bc50 con 0x7f507408a410 2024-09-16T10:50:04.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e380373f0 0x7f3e380398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f3e3c048b50 con 0x7f3e4c13f6a0 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 client.4284.objecter ms_dispatch 0x7f3e4c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5078e3a700 1 --2- 172.21.15.5:0/824558422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507408a410 0x7f507408a7f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f507409faf0 tx=0x7f506800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44dda5c6d3e7cc1d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 3 client.4284.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 3 client.4284.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 20 client.4284.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 1 -- 172.21.15.5:0/824558422 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506800ddd0 con 0x7f507408a410 2024-09-16T10:50:04.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51e43700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e380373f0 0x7f3e380398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: dump: 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 5 monclient: authenticate success, global_id 4287 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 1 -- 172.21.15.5:0/824558422 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068016440 con 0x7f507408a410 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 1 -- 172.21.15.5:0/824558422 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680167d0 con 0x7f507408a410 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 4 set_mon_vals no callback set 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f50737fe700 10 monclient: dump: 2024-09-16T10:50:04.720 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51e43700 10 monclient: get_auth_request con 0x7f3e380373f0 auth_method 0 2024-09-16T10:50:04.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e51e43700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e380373f0 0x7f3e380398b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e440012e0 tx=0x7f3e44001420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f3e427fc700 10 client.4284.objecter ms_handle_connect 0x7f3e380373f0 2024-09-16T10:50:04.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 10 monclient: shutdown 2024-09-16T10:50:04.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 -- 172.21.15.5:0/824558422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507408a410 msgr2=0x7f507408a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.612+0000 7f5079e3c700 1 --2- 172.21.15.5:0/824558422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507408a410 0x7f507408a7f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f507409faf0 tx=0x7f506800bde0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/824558422 shutdown_connections 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 --2- 172.21.15.5:0/824558422 >> v2:172.21.15.143:3300/0 conn(0x7f507409f400 0x7f507414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 --2- 172.21.15.5:0/824558422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507408a410 0x7f507408a7f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/824558422 >> 172.21.15.5:0/824558422 conn(0x7f50740a45d0 msgr2=0x7f50740a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/824558422 shutdown_connections 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/824558422 wait complete. 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f4035e03700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4014036f60 0x7f4014039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f4035e03700 10 monclient: get_auth_request con 0x7f4014036f60 auth_method 0 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: monmap: 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.724 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.725 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f4035e03700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4014036f60 0x7f4014039420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f402c0015a0 tx=0x7f402c007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f40267fc700 10 client.4278.objecter ms_handle_connect 0x7f4014036f60 2024-09-16T10:50:04.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 librados: starting msgr at 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 librados: starting objecter 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 Processor -- start 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- start start 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 librados: setting wanted keys 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 librados: calling monclient init 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: init 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f507408a410 0x7f50741d7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-b con 0x7f507408a410 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 0x7f50741d7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-c con 0x7f507409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 0x7f507407daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): picked mon.noname-a con 0x7f507409f400 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5074081770 con 0x7f507409f400 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50740802c0 con 0x7f507408a410 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50740a2680 con 0x7f507409e8e0 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient(hunting): authenticate will time out at 863.197818s 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 0x7f50741d7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 0x7f507407daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 0x7f50741d7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37922/0 (socket says 172.21.15.5:37922) 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 0x7f507407daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48860/0 (socket says 172.21.15.5:48860) 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 -- 172.21.15.5:0/1315822355 learned_addr learned my addr 172.21.15.5:0/1315822355 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): get_auth_request con 0x7f507409f400 auth_method 0 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): get_auth_request con 0x7f507409e8e0 auth_method 0 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f507408a410 0x7f50741d7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): get_auth_request con 0x7f507408a410 auth_method 0 2024-09-16T10:50:04.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5078e3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-16T10:50:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 -- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 msgr2=0x7f50741d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 0x7f50741d7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 -- 172.21.15.5:0/1315822355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f507408a410 msgr2=0x7f50741d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f507408a410 0x7f50741d7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 -- 172.21.15.5:0/1315822355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506800b760 con 0x7f507409f400 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5073fff700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f507409e8e0 0x7f50741d7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f507963b700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 0x7f507407daf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f506c00b970 tx=0x7f506c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 client.?.objecter ms_handle_connect 0x7f507409f400 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00d860 con 0x7f507409f400 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: dump: 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00d9e0 con 0x7f507409f400 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 5 monclient: authenticate success, global_id 4290 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _renew_subs 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5074071000 con 0x7f507409f400 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00ddd0 con 0x7f507409f400 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f50717fa700 10 monclient: dump: 2024-09-16T10:50:04.735 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _renew_subs 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5074081490 con 0x7f507409f400 2024-09-16T10:50:04.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 librados: init done 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 10 client.4281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 15 client.4281.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6804/3538568827 conn(0x7f1ddc11ec10 0x7f1ddc11eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 20 client.4281.objecter _get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 20 client.4281.objecter _get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 15 client.4281.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 20 client.4281.objecter get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 15 client.4281.objecter _session_command_op_assign 5 1 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 4 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 10 client.4281.objecter _send_command 1 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 --> v2:172.21.15.143:6804/3538568827 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ddc071940 con 0x7f1ddc11ec10 2024-09-16T10:50:04.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5072ffd700 4 set_mon_vals no callback set 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 15 client.4290.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 20 client.4290.objecter _get_session osd=-1 returning homeless 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.616+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50540028b0 con 0x7f507409f400 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f506c00e430 con 0x7f507409f400 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5058036f60 0x7f5058039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f506c047970 con 0x7f507409f400 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter ms_dispatch 0x7f5074000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 3 client.4290.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 3 client.4290.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.740 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:04.741 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:50:04.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter checking command 1 2024-09-16T10:50:04.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.5:6804/2015722671 conn(0x7f505803ca10 0x7f505803ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter _get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 15 client.4290.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter _get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 15 client.4290.objecter _session_command_op_assign 2 1 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 4 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter _send_command 1 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 --> v2:172.21.15.5:6804/2015722671 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f505803f4e0 con 0x7f505803ca10 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f506c04c2a0 con 0x7f507409f400 2024-09-16T10:50:04.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1de2993700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6804/3538568827 conn(0x7f1ddc11ec10 0x7f1ddc11eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1de2993700 10 monclient: get_auth_request con 0x7f1ddc11ec10 auth_method 0 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5078e3a700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5058036f60 0x7f5058039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5078e3a700 10 monclient: get_auth_request con 0x7f5058036f60 auth_method 0 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5073fff700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.5:6804/2015722671 conn(0x7f505803ca10 0x7f505803ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5073fff700 10 monclient: get_auth_request con 0x7f505803ca10 auth_method 0 2024-09-16T10:50:04.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 10 client.4281.objecter ms_handle_connect 0x7f1ddc11ec10 2024-09-16T10:50:04.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1de2993700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6804/3538568827 conn(0x7f1ddc11ec10 0x7f1ddc11eff0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 10 client.4284.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_assign -1 1 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6800/3342511476 conn(0x7f3e34002f60 0x7f3e34005420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 20 client.4284.objecter _get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 20 client.4284.objecter _get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_remove -1 1 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 20 client.4284.objecter get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_assign 3 1 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 4 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 10 client.4284.objecter _send_command 1 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 --> v2:172.21.15.143:6800/3342511476 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e34008480 con 0x7f3e34002f60 2024-09-16T10:50:04.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter ms_handle_connect 0x7f505803ca10 2024-09-16T10:50:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5073fff700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.5:6804/2015722671 conn(0x7f505803ca10 0x7f505803ee50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e51642700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6800/3342511476 conn(0x7f3e34002f60 0x7f3e34005420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e51642700 10 monclient: get_auth_request con 0x7f3e34002f60 auth_method 0 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter ms_handle_connect 0x7f5058036f60 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f5078e3a700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5058036f60 0x7f5058039420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5068000c00 tx=0x7f5068012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== osd.2 v2:172.21.15.5:6804/2015722671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f505803f4e0 con 0x7f505803ca10 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter ms_dispatch 0x7f5074000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 10 client.4290.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 4 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f50717fa700 15 client.4290.objecter _session_command_op_remove 2 1 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 10 client.4284.objecter ms_handle_connect 0x7f3e34002f60 2024-09-16T10:50:04.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e51642700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6800/3342511476 conn(0x7f3e34002f60 0x7f3e34005420 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== osd.5 v2:172.21.15.143:6804/3538568827 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f1dcc006880 con 0x7f1ddc11ec10 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 10 client.4281.objecter ms_dispatch 0x7f1ddc000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 10 client.4281.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 4 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f1dd2ffd700 15 client.4281.objecter _session_command_op_remove 5 1 2024-09-16T10:50:04.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== osd.3 v2:172.21.15.143:6800/3342511476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3e34008480 con 0x7f3e34002f60 2024-09-16T10:50:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 10 client.4284.objecter ms_dispatch 0x7f3e4c000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T10:50:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 10 client.4284.objecter _finish_command 1 = system:0 2024-09-16T10:50:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 4 2024-09-16T10:50:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.620+0000 7f3e427fc700 15 client.4284.objecter _session_command_op_remove 3 1 2024-09-16T10:50:04.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 client.4278.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 15 client.4278.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter _get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter _get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 15 client.4278.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter get_session s=0x7f401403c360 osd=1 3 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 15 client.4278.objecter _session_command_op_assign 1 2 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 4 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 client.4278.objecter _send_command 2 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 --> v2:172.21.15.5:6800/3675136489 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40300741c0 con 0x7f401403ca10 2024-09-16T10:50:04.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f40267fc700 1 -- 172.21.15.5:0/3261605638 <== osd.1 v2:172.21.15.5:6800/3675136489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40300741c0 con 0x7f401403ca10 2024-09-16T10:50:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f40267fc700 10 client.4278.objecter ms_dispatch 0x7f4030000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f40267fc700 10 client.4278.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f40267fc700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 4 2024-09-16T10:50:04.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f40267fc700 15 client.4278.objecter _session_command_op_remove 1 2 2024-09-16T10:50:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 librados: watch_flush enter 2024-09-16T10:50:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 librados: watch_flush exit 2024-09-16T10:50:04.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 client.4278.objecter close_session for osd.1 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 >> v2:172.21.15.5:6800/3675136489 conn(0x7f401403ca10 msgr2=0x7f401403ee50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.5:6800/3675136489 conn(0x7f401403ca10 0x7f401403ee50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter put_session s=0x7f401403c360 osd=1 1 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 client.4278.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 client.4278.objecter successfully canceled tick 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4014036f60 msgr2=0x7f4014039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 --2- 172.21.15.5:0/3261605638 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4014036f60 0x7f4014039420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f402c0015a0 tx=0x7f402c007a80 comp rx=0 tx=0).stop 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 10 monclient: shutdown 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 msgr2=0x7f4030124290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 --2- 172.21.15.5:0/3261605638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f403013b0e0 0x7f4030124290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f402800efc0 tx=0x7f402800c3a0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036604700 1 -- 172.21.15.5:0/3261605638 reap_dead start 2024-09-16T10:50:04.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 shutdown_connections 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 >> 172.21.15.5:0/3261605638 conn(0x7f403006a4f0 msgr2=0x7f403009e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 shutdown_connections 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 -- 172.21.15.5:0/3261605638 wait complete. 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.628+0000 7f4036e05700 1 librados: shutdown 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 client.4281.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 15 client.4281.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter _get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter _get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 15 client.4281.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter get_session s=0x7f1ddc136550 osd=5 3 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 15 client.4281.objecter _session_command_op_assign 5 2 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 4 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 client.4281.objecter _send_command 2 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 --> v2:172.21.15.143:6804/3538568827 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ddc0a1360 con 0x7f1ddc11ec10 2024-09-16T10:50:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 15 client.4290.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter _get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter _get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 15 client.4290.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter get_session s=0x7f505803c360 osd=2 3 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 15 client.4290.objecter _session_command_op_assign 2 2 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 4 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 client.4290.objecter _send_command 2 2024-09-16T10:50:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 --> v2:172.21.15.5:6804/2015722671 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50540011a0 con 0x7f505803ca10 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1dd2ffd700 1 -- 172.21.15.5:0/896493289 <== osd.5 v2:172.21.15.143:6804/3538568827 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1dcc005090 con 0x7f1ddc11ec10 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1dd2ffd700 10 client.4281.objecter ms_dispatch 0x7f1ddc000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1dd2ffd700 10 client.4281.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1dd2ffd700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 4 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1dd2ffd700 15 client.4281.objecter _session_command_op_remove 5 2 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f50717fa700 1 -- 172.21.15.5:0/1315822355 <== osd.2 v2:172.21.15.5:6804/2015722671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f50540011a0 con 0x7f505803ca10 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f50717fa700 10 client.4290.objecter ms_dispatch 0x7f5074000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f50717fa700 10 client.4290.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f50717fa700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 4 2024-09-16T10:50:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f50717fa700 15 client.4290.objecter _session_command_op_remove 2 2 2024-09-16T10:50:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 librados: watch_flush enter 2024-09-16T10:50:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 librados: watch_flush exit 2024-09-16T10:50:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 client.4281.objecter close_session for osd.5 2024-09-16T10:50:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6804/3538568827 conn(0x7f1ddc11ec10 msgr2=0x7f1ddc11eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 librados: watch_flush enter 2024-09-16T10:50:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6804/3538568827 conn(0x7f1ddc11ec10 0x7f1ddc11eff0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 librados: watch_flush exit 2024-09-16T10:50:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 client.4290.objecter close_session for osd.2 2024-09-16T10:50:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter put_session s=0x7f1ddc136550 osd=5 1 2024-09-16T10:50:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 client.4281.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 >> v2:172.21.15.5:6804/2015722671 conn(0x7f505803ca10 msgr2=0x7f505803ee50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.5:6804/2015722671 conn(0x7f505803ca10 0x7f505803ee50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 client.4281.objecter successfully canceled tick 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter put_session s=0x7f505803c360 osd=2 1 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 client.4290.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 client.4290.objecter successfully canceled tick 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1dc0036f60 msgr2=0x7f1dc0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 --2- 172.21.15.5:0/896493289 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1dc0036f60 0x7f1dc0039420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007d00 tx=0x7f1dd8007c50 comp rx=0 tx=0).stop 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 10 monclient: shutdown 2024-09-16T10:50:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ddc070e60 msgr2=0x7f1ddc1346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 --2- 172.21.15.5:0/896493289 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ddc070e60 0x7f1ddc1346c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4014000 tx=0x7f1dd4004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5058036f60 msgr2=0x7f5058039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 --2- 172.21.15.5:0/1315822355 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5058036f60 0x7f5058039420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5068000c00 tx=0x7f5068012000 comp rx=0 tx=0).stop 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 10 monclient: shutdown 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 msgr2=0x7f507407daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 --2- 172.21.15.5:0/1315822355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f507409f400 0x7f507407daf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f506c00b970 tx=0x7f506c00bc80 comp rx=0 tx=0).stop 2024-09-16T10:50:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f507963b700 1 -- 172.21.15.5:0/1315822355 reap_dead start 2024-09-16T10:50:04.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de2993700 1 -- 172.21.15.5:0/896493289 reap_dead start 2024-09-16T10:50:04.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 shutdown_connections 2024-09-16T10:50:04.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 >> 172.21.15.5:0/1315822355 conn(0x7f50740a45d0 msgr2=0x7f507409cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 shutdown_connections 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 >> 172.21.15.5:0/896493289 conn(0x7f1ddc0a45d0 msgr2=0x7f1ddc1bd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 shutdown_connections 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 -- 172.21.15.5:0/896493289 wait complete. 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f1de3194700 1 librados: shutdown 2024-09-16T10:50:04.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 shutdown_connections 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 -- 172.21.15.5:0/1315822355 wait complete. 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.636+0000 7f5079e3c700 1 librados: shutdown 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 client.4284.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_assign -1 2 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter _get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter _get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_remove -1 2 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter get_session s=0x7f3e34002870 osd=3 3 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 15 client.4284.objecter _session_command_op_assign 3 2 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 4 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 client.4284.objecter _send_command 2 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 --> v2:172.21.15.143:6800/3342511476 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e340011a0 con 0x7f3e34002f60 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e427fc700 1 -- 172.21.15.5:0/2647981288 <== osd.3 v2:172.21.15.143:6800/3342511476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3e340011a0 con 0x7f3e34002f60 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e427fc700 10 client.4284.objecter ms_dispatch 0x7f3e4c000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e427fc700 10 client.4284.objecter _finish_command 2 = system:0 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e427fc700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 4 2024-09-16T10:50:04.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e427fc700 15 client.4284.objecter _session_command_op_remove 3 2 2024-09-16T10:50:04.767 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:04.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 librados: watch_flush enter 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 librados: watch_flush exit 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 client.4284.objecter close_session for osd.3 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6800/3342511476 conn(0x7f3e34002f60 msgr2=0x7f3e34005420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6800/3342511476 conn(0x7f3e34002f60 0x7f3e34005420 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter put_session s=0x7f3e34002870 osd=3 1 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 client.4284.objecter successfully canceled tick 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e380373f0 msgr2=0x7f3e380398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2647981288 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e380373f0 0x7f3e380398b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e440012e0 tx=0x7f3e44001420 comp rx=0 tx=0).stop 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 10 monclient: shutdown 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 msgr2=0x7f3e4c122d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 --2- 172.21.15.5:0/2647981288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e4c13f6a0 0x7f3e4c122d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c016000 tx=0x7f3e3c004770 comp rx=0 tx=0).stop 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52644700 1 -- 172.21.15.5:0/2647981288 reap_dead start 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 shutdown_connections 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 >> 172.21.15.5:0/2647981288 conn(0x7f3e4c0a45d0 msgr2=0x7f3e4c13da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 shutdown_connections 2024-09-16T10:50:04.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 -- 172.21.15.5:0/2647981288 wait complete. 2024-09-16T10:50:04.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.644+0000 7f3e52e45700 1 librados: shutdown 2024-09-16T10:50:04.771 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-16T10:50:04.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T10:50:04.772 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:04.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:50:04.773 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:04.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T10:50:04.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 10 monclient: monmap: 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.923469+0000 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.923469+0000 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.927 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.927 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 1 Processor -- start 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 1 -- start start 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.916+0000 7f3887fff700 10 monclient: init 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3888070ad0 0x7f3888070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-a con 0x7f3888070ad0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f388809dc30 0x7f38880a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-c con 0x7f388809dc30 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38880713f0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388809e0b0 con 0x7f38880713f0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388809f150 con 0x7f3888070ad0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388809d050 con 0x7f388809dc30 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient(hunting): authenticate will time out at 863.500472s 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3888070ad0 0x7f3888070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3888070ad0 0x7f3888070eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48864/0 (socket says 172.21.15.5:48864) 2024-09-16T10:50:04.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 -- 172.21.15.5:0/33975275 learned_addr learned my addr 172.21.15.5:0/33975275 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): get_auth_request con 0x7f3888070ad0 auth_method 0 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f38867fc700 1 --2- 172.21.15.5:0/33975275 >> v2:172.21.15.143:3300/0 conn(0x7f388809dc30 0x7f38880a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f38867fc700 10 monclient(hunting): get_auth_request con 0x7f388809dc30 auth_method 0 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f38867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f38867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f38867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 -- 172.21.15.5:0/33975275 >> v2:172.21.15.143:3300/0 conn(0x7f388809dc30 msgr2=0x7f38880a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 --2- 172.21.15.5:0/33975275 >> v2:172.21.15.143:3300/0 conn(0x7f388809dc30 0x7f38880a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 -- 172.21.15.5:0/33975275 >> v1:172.21.15.143:6789/0 conn(0x7f38880713f0 legacy=0x7f38880a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 monclient: found mon.noname-a 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 -- 172.21.15.5:0/33975275 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388809f9f0 con 0x7f3888070ad0 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3886ffd700 1 --2- 172.21.15.5:0/33975275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f3888070eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3888074b90 tx=0x7f3878009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6db6c7d241fb42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 1 -- 172.21.15.5:0/33975275 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800b8a0 con 0x7f3888070ad0 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:04.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: dump: 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 5 monclient: authenticate success, global_id 4293 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 1 -- 172.21.15.5:0/33975275 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387800ba00 con 0x7f3888070ad0 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 1 -- 172.21.15.5:0/33975275 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387800bd90 con 0x7f3888070ad0 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3885ffb700 10 monclient: dump: 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.934 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 4 set_mon_vals no callback set 2024-09-16T10:50:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 10 monclient: shutdown 2024-09-16T10:50:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- 172.21.15.5:0/33975275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 msgr2=0x7f3888070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 --2- 172.21.15.5:0/33975275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f3888070eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3888074b90 tx=0x7f3878009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- 172.21.15.5:0/33975275 shutdown_connections 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 --2- 172.21.15.5:0/33975275 >> v2:172.21.15.143:3300/0 conn(0x7f388809dc30 0x7f38880a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 --2- 172.21.15.5:0/33975275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f3888070eb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- 172.21.15.5:0/33975275 >> 172.21.15.5:0/33975275 conn(0x7f38880a3200 msgr2=0x7f3888073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- 172.21.15.5:0/33975275 shutdown_connections 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.920+0000 7f3887fff700 1 -- 172.21.15.5:0/33975275 wait complete. 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: monmap: 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 librados: starting msgr at 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 librados: starting objecter 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 Processor -- start 2024-09-16T10:50:04.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- start start 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 librados: setting wanted keys 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 librados: calling monclient init 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: init 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f38881c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-a con 0x7f3888070ad0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-c con 0x7f38880713f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 0x7f38881ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): picked mon.noname-b con 0x7f38881ca510 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38881ab620 con 0x7f3888070ad0 2024-09-16T10:50:04.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388809d1d0 con 0x7f38881ca510 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388806d820 con 0x7f38880713f0 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient(hunting): authenticate will time out at 863.505767s 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f38881c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37930/0 (socket says 172.21.15.5:37930) 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 -- 172.21.15.5:0/1745027948 learned_addr learned my addr 172.21.15.5:0/1745027948 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): get_auth_request con 0x7f38880713f0 auth_method 0 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): get_auth_request con 0x7f3888070ad0 auth_method 0 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38877fe700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 0x7f38881ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 -- 172.21.15.5:0/1745027948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 msgr2=0x7f38881ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 0x7f38881ca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 -- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 msgr2=0x7f38881c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f38881c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 monclient: found mon.noname-c 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 -- 172.21.15.5:0/1745027948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878009570 con 0x7f38880713f0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38877fe700 10 monclient: get_auth_request con 0x7f38881ca510 auth_method 0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38877fe700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 0x7f38881ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3886ffd700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f38881c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38867fc700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f387c00e790 tx=0x7f387c00eaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 client.?.objecter ms_handle_connect 0x7f38880713f0 2024-09-16T10:50:04.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c013610 con 0x7f38880713f0 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: dump: 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 5 monclient: authenticate success, global_id 4247 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c013770 con 0x7f38880713f0 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _renew_subs 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38880a2350 con 0x7f38880713f0 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c013ae0 con 0x7f38880713f0 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f386ffff700 10 monclient: dump: 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.948 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f38857fa700 4 set_mon_vals no callback set 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _renew_subs 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38881abec0 con 0x7f38880713f0 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.924+0000 7f3887fff700 1 librados: init done 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f387c013c40 con 0x7f38880713f0 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 1 --2- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 0x7f3874041c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f387c047300 con 0x7f38880713f0 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 10 client.4247.objecter ms_dispatch 0x7f3888000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 3 client.4247.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 3 client.4247.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 20 client.4247.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f3886ffd700 1 --2- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 0x7f3874041c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f3886ffd700 10 monclient: get_auth_request con 0x7f387403f770 auth_method 0 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f386ffff700 10 client.4247.objecter ms_handle_connect 0x7f387403f770 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.928+0000 7f3886ffd700 1 --2- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 0x7f3874041c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3878000c00 tx=0x7f3878008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f3887fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f3887fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f3887fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38880a1b10 con 0x7f38880713f0 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 10 monclient: monmap: 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.939285+0000 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.939285+0000 2024-09-16T10:50:04.955 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 1 Processor -- start 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 1 -- start start 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.932+0000 7f544f5cc700 10 monclient: init 2024-09-16T10:50:04.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5448135f10 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5448071320 0x7f544813acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-c con 0x7f5448071320 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f544813b200 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54481359a0 con 0x7f544813b200 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480a09a0 con 0x7f5448135f10 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544809fdc0 con 0x7f5448071320 2024-09-16T10:50:04.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient(hunting): authenticate will time out at 863.516391s 2024-09-16T10:50:04.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544db69700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f544813b200 0x7f544813d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58986/0 (socket says 172.21.15.5:58986) 2024-09-16T10:50:04.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544db69700 1 -- 172.21.15.5:0/4084394693 learned_addr learned my addr 172.21.15.5:0/4084394693 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 --2- 172.21.15.5:0/4084394693 >> v2:172.21.15.143:3300/0 conn(0x7f5448071320 0x7f544813acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): get_auth_request con 0x7f5448071320 auth_method 0 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 1 -- 172.21.15.5:0/4084394693 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824256533 0 0) 0x7f54481359a0 con 0x7f544813b200 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient(hunting): _negotiate 2024-09-16T10:50:04.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient(hunting): my global_id is 4299 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 1 -- 172.21.15.5:0/4084394693 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54340035d0 con 0x7f544813b200 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 -- 172.21.15.5:0/4084394693 >> v1:172.21.15.143:6789/0 conn(0x7f5448135f10 legacy=0x7f54481362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 -- 172.21.15.5:0/4084394693 >> v1:172.21.15.5:6789/0 conn(0x7f544813b200 legacy=0x7f544813d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 monclient: found mon.noname-c 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 -- 172.21.15.5:0/4084394693 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480a07a0 con 0x7f5448071320 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544cb67700 1 --2- 172.21.15.5:0/4084394693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448071320 0x7f544813acc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54380097d0 tx=0x7f5438009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9d191ea94792bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 1 -- 172.21.15.5:0/4084394693 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438004230 con 0x7f5448071320 2024-09-16T10:50:04.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: dump: 2024-09-16T10:50:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.964 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 5 monclient: authenticate success, global_id 4297 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 1 -- 172.21.15.5:0/4084394693 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5438004390 con 0x7f5448071320 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 1 -- 172.21.15.5:0/4084394693 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54380036c0 con 0x7f5448071320 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 4 set_mon_vals no callback set 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f543ffff700 10 monclient: dump: 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.966 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 10 monclient: shutdown 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- 172.21.15.5:0/4084394693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448071320 msgr2=0x7f544813acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 --2- 172.21.15.5:0/4084394693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448071320 0x7f544813acc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f54380097d0 tx=0x7f5438009ae0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- 172.21.15.5:0/4084394693 shutdown_connections 2024-09-16T10:50:04.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 --2- 172.21.15.5:0/4084394693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448071320 0x7f544813acc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- 172.21.15.5:0/4084394693 >> 172.21.15.5:0/4084394693 conn(0x7f54480a0b30 msgr2=0x7f54480a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- 172.21.15.5:0/4084394693 shutdown_connections 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.936+0000 7f544f5cc700 1 -- 172.21.15.5:0/4084394693 wait complete. 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient: monmap: 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 librados: starting msgr at 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 librados: starting objecter 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 Processor -- start 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 -- start start 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 librados: setting wanted keys 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 librados: calling monclient init 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient: init 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448135f10 0x7f5448139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-b con 0x7f5448135f10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 0x7f54481309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-a con 0x7f544813b200 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): picked mon.noname-c con 0x7f5448130f30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544811cde0 con 0x7f544813b200 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544809c920 con 0x7f5448135f10 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544809b1b0 con 0x7f5448130f30 2024-09-16T10:50:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544f5cc700 10 monclient(hunting): authenticate will time out at 863.521790s 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37936/0 (socket says 172.21.15.5:37936) 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 -- 172.21.15.5:0/1903236750 learned_addr learned my addr 172.21.15.5:0/1903236750 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): get_auth_request con 0x7f5448130f30 auth_method 0 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448135f10 0x7f5448139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 0x7f54481309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): get_auth_request con 0x7f5448135f10 auth_method 0 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f387c012370 con 0x7f38880713f0 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f386ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f386ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 10 monclient(hunting): get_auth_request con 0x7f544813b200 auth_method 0 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544d368700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 -- 172.21.15.5:0/1903236750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448135f10 msgr2=0x7f5448139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448135f10 0x7f5448139990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 -- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 msgr2=0x7f54481309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 0x7f54481309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 monclient: found mon.noname-c 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544db69700 1 -- 172.21.15.5:0/1903236750 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380094b0 con 0x7f5448130f30 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.940+0000 7f544cb67700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 0x7f54481309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544db69700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5440002230 tx=0x7f544000cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 client.?.objecter ms_handle_connect 0x7f5448130f30 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544000ab70 con 0x7f5448130f30 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: dump: 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544000acf0 con 0x7f5448130f30 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440007310 con 0x7f5448130f30 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 monclient: dump: 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543effd700 4 set_mon_vals no callback set 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 5 monclient: authenticate success, global_id 4250 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 10 monclient: _renew_subs 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544811d3a0 con 0x7f5448130f30 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 10 monclient: _renew_subs 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544813a580 con 0x7f5448130f30 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544f5cc700 1 librados: init done 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5440007470 con 0x7f5448130f30 2024-09-16T10:50:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 --2- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 0x7f542c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f54400471d0 con 0x7f5448130f30 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 10 client.4250.objecter ms_dispatch 0x7f5448000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 3 client.4250.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 3 client.4250.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f543dffb700 20 client.4250.objecter dump_active .. 0 homeless 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544d368700 1 --2- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 0x7f542c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.944+0000 7f544d368700 10 monclient: get_auth_request con 0x7f542c036f60 auth_method 0 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.948+0000 7f543dffb700 10 client.4250.objecter ms_handle_connect 0x7f542c036f60 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.948+0000 7f544d368700 1 --2- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 0x7f542c039420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5444001a80 tx=0x7f5444009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.952+0000 7f544f5cc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.952+0000 7f544f5cc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.952+0000 7f544f5cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.952+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54481362f0 con 0x7f5448130f30 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: monmap: 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.960985+0000 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.960985+0000 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 Processor -- start 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 -- start start 2024-09-16T10:50:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: init 2024-09-16T10:50:04.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5440018080 con 0x7f5448130f30 2024-09-16T10:50:04.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f543dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:04.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f543dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a88135c30 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a8813aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-b con 0x7f9a880a04f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a8813afb0 0x7f9a8813d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-c con 0x7f9a8813afb0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a88135750 con 0x7f9a88135c30 2024-09-16T10:50:04.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a8809f830 con 0x7f9a8813afb0 2024-09-16T10:50:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8809f1c0 con 0x7f9a880a04f0 2024-09-16T10:50:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8df2b700 10 monclient(hunting): authenticate will time out at 863.539548s 2024-09-16T10:50:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f6331ce4700 10 monclient: get_monmap_and_config 2024-09-16T10:50:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f6331ce4700 10 monclient: build_initial_monmap 2024-09-16T10:50:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f6331ce4700 10 monclient: monmap: 2024-09-16T10:50:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.963805+0000 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.963805+0000 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:04.990 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f6331ce4700 1 Processor -- start 2024-09-16T10:50:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8cf29700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9a88135c30 0x7f9a88136010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59000/0 (socket says 172.21.15.5:59000) 2024-09-16T10:50:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8cf29700 1 -- 172.21.15.5:0/3468037752 learned_addr learned my addr 172.21.15.5:0/3468037752 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a87fff700 1 --2- 172.21.15.5:0/3468037752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a8813aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a87fff700 10 monclient(hunting): get_auth_request con 0x7f9a880a04f0 auth_method 0 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.956+0000 7f9a8d72a700 1 --2- 172.21.15.5:0/3468037752 >> v2:172.21.15.143:3300/0 conn(0x7f9a8813afb0 0x7f9a8813d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): get_auth_request con 0x7f9a8813afb0 auth_method 0 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8d72a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 -- 172.21.15.5:0/3468037752 >> v2:172.21.15.143:3300/0 conn(0x7f9a8813afb0 msgr2=0x7f9a8813d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 --2- 172.21.15.5:0/3468037752 >> v2:172.21.15.143:3300/0 conn(0x7f9a8813afb0 0x7f9a8813d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 -- 172.21.15.5:0/3468037752 >> v1:172.21.15.5:6789/0 conn(0x7f9a88135c30 legacy=0x7f9a88136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 monclient: found mon.noname-b 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 -- 172.21.15.5:0/3468037752 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88071b30 con 0x7f9a880a04f0 2024-09-16T10:50:04.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a87fff700 1 --2- 172.21.15.5:0/3468037752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a8813aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009d90 tx=0x7f9a8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4e5a2e04ccf33e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 -- start start 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient: init 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f632c0a10f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c132110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-b con 0x7f632c09b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f632c136ba0 0x7f632c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-c con 0x7f632c136ba0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 1 -- 172.21.15.5:0/3468037752 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000e070 con 0x7f9a880a04f0 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: dump: 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.996 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 1 -- 172.21.15.5:0/3468037752 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80003a40 con 0x7f9a880a04f0 2024-09-16T10:50:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 1 -- 172.21.15.5:0/3468037752 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003df0 con 0x7f9a880a04f0 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a877fe700 10 monclient: dump: 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:04.998 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:04.999 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:04.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 5 monclient: authenticate success, global_id 4253 2024-09-16T10:50:04.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:04.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 4 set_mon_vals no callback set 2024-09-16T10:50:05.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c09c0f0 con 0x7f632c0a10f0 2024-09-16T10:50:05.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c09fc70 con 0x7f632c136ba0 2024-09-16T10:50:05.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0a14d0 con 0x7f632c09b3e0 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient(hunting): authenticate will time out at 863.541165s 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6330ce2700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f632c0a10f0 0x7f632c131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59004/0 (socket says 172.21.15.5:59004) 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6330ce2700 1 -- 172.21.15.5:0/2177182444 learned_addr learned my addr 172.21.15.5:0/2177182444 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c132110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): get_auth_request con 0x7f632c09b3e0 auth_method 0 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 --2- 172.21.15.5:0/2177182444 >> v2:172.21.15.143:3300/0 conn(0x7f632c136ba0 0x7f632c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): get_auth_request con 0x7f632c136ba0 auth_method 0 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 -- 172.21.15.5:0/2177182444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 msgr2=0x7f632c132110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c132110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 -- 172.21.15.5:0/2177182444 >> v1:172.21.15.5:6789/0 conn(0x7f632c0a10f0 legacy=0x7f632c131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 monclient: found mon.noname-c 2024-09-16T10:50:05.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:50:05.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 -- 172.21.15.5:0/2177182444 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c09be10 con 0x7f632c136ba0 2024-09-16T10:50:05.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 10 monclient: shutdown 2024-09-16T10:50:05.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- 172.21.15.5:0/3468037752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 msgr2=0x7f9a8813aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/3468037752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a8813aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009d90 tx=0x7f9a8000b330 comp rx=0 tx=0).stop 2024-09-16T10:50:05.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f63314e3700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c136ba0 0x7f632c138f90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f631c009850 tx=0x7f631c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a07c7f3e991c7560 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- 172.21.15.5:0/3468037752 shutdown_connections 2024-09-16T10:50:05.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/3468037752 >> v2:172.21.15.143:3300/0 conn(0x7f9a8813afb0 0x7f9a8813d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/3468037752 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a8813aa70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- 172.21.15.5:0/3468037752 >> 172.21.15.5:0/3468037752 conn(0x7f9a880a45d0 msgr2=0x7f9a880a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- 172.21.15.5:0/3468037752 shutdown_connections 2024-09-16T10:50:05.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- 172.21.15.5:0/3468037752 wait complete. 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 1 -- 172.21.15.5:0/2177182444 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c011830 con 0x7f632c136ba0 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: dump: 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.007 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 5 monclient: authenticate success, global_id 4309 2024-09-16T10:50:05.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:05.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 1 -- 172.21.15.5:0/2177182444 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c011990 con 0x7f632c136ba0 2024-09-16T10:50:05.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f632b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:05.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f6331ce4700 4 set_mon_vals no callback set 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 10 monclient: monmap: 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:05.010 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 librados: starting msgr at 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 librados: starting objecter 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 Processor -- start 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 -- start start 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 librados: setting wanted keys 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 1 librados: calling monclient init 2024-09-16T10:50:05.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.960+0000 7f9a8df2b700 10 monclient: init 2024-09-16T10:50:05.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632b7fe700 1 -- 172.21.15.5:0/2177182444 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c011d20 con 0x7f632c136ba0 2024-09-16T10:50:05.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:05.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632b7fe700 10 monclient: dump: 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.013 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: shutdown 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- 172.21.15.5:0/2177182444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c136ba0 msgr2=0x7f632c138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c136ba0 0x7f632c138f90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f631c009850 tx=0x7f631c009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- 172.21.15.5:0/2177182444 shutdown_connections 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c136ba0 0x7f632c138f90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- 172.21.15.5:0/2177182444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c132110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- 172.21.15.5:0/2177182444 >> 172.21.15.5:0/2177182444 conn(0x7f632c0a45d0 msgr2=0x7f632c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- 172.21.15.5:0/2177182444 shutdown_connections 2024-09-16T10:50:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- 172.21.15.5:0/2177182444 wait complete. 2024-09-16T10:50:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a881ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-c con 0x7f9a880a04f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-a con 0x7f9a88135c30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a8813afb0 0x7f9a8811b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): picked mon.noname-b con 0x7f9a8813afb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8811f0b0 con 0x7f9a88135c30 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8811f300 con 0x7f9a8813afb0 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8809b0d0 con 0x7f9a880a04f0 2024-09-16T10:50:05.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8df2b700 10 monclient(hunting): authenticate will time out at 863.544528s 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a881ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48898/0 (socket says 172.21.15.5:48898) 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 -- 172.21.15.5:0/1444913320 learned_addr learned my addr 172.21.15.5:0/1444913320 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): get_auth_request con 0x7f9a88135c30 auth_method 0 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): get_auth_request con 0x7f9a880a04f0 auth_method 0 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8cf29700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 -- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 msgr2=0x7f9a881ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a881ca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 -- 172.21.15.5:0/1444913320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a8813afb0 msgr2=0x7f9a8811b340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a8d72a700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a8813afb0 0x7f9a8811b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a8813afb0 0x7f9a8811b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 monclient: found mon.noname-a 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 -- 172.21.15.5:0/1444913320 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a80009a70 con 0x7f9a88135c30 2024-09-16T10:50:05.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: monmap: 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:05.020 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f9a87fff700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000b370 tx=0x7f9a80004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 librados: starting msgr at 2024-09-16T10:50:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 librados: starting objecter 2024-09-16T10:50:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 Processor -- start 2024-09-16T10:50:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- start start 2024-09-16T10:50:05.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 librados: setting wanted keys 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 librados: calling monclient init 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: init 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c1c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-c con 0x7f632c09b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c0a10f0 0x7f632c12da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-b con 0x7f632c0a10f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 0x7f632c12df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): picked mon.noname-a con 0x7f632c136ba0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1c6560 con 0x7f632c136ba0 2024-09-16T10:50:05.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c071070 con 0x7f632c0a10f0 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c06fbc0 con 0x7f632c09b3e0 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6331ce4700 10 monclient(hunting): authenticate will time out at 863.546331s 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c1c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c1c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37970/0 (socket says 172.21.15.5:37970) 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 1 -- 172.21.15.5:0/1116555497 learned_addr learned my addr 172.21.15.5:0/1116555497 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): get_auth_request con 0x7f632c09b3e0 auth_method 0 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 0x7f632c12df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): get_auth_request con 0x7f632c136ba0 auth_method 0 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c0a10f0 0x7f632c12da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): get_auth_request con 0x7f632c0a10f0 auth_method 0 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f6330ce2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f632bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 -- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 msgr2=0x7f632c1c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c1c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 -- 172.21.15.5:0/1116555497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c0a10f0 msgr2=0x7f632c12da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c0a10f0 0x7f632c12da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 monclient: found mon.noname-a 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 -- 172.21.15.5:0/1116555497 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c0094b0 con 0x7f632c136ba0 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.964+0000 7f63314e3700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 0x7f632c12df80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f631c016000 tx=0x7f631c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 client.?.objecter ms_handle_connect 0x7f632c136ba0 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:05.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c006ea0 con 0x7f632c136ba0 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: dump: 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.031 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 5 monclient: authenticate success, global_id 4314 2024-09-16T10:50:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c01f070 con 0x7f632c136ba0 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _renew_subs 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c0a2230 con 0x7f632c136ba0 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c004610 con 0x7f632c136ba0 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 monclient: dump: 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.033 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f632a7fc700 4 set_mon_vals no callback set 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _renew_subs 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c06f940 con 0x7f632c136ba0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 1 librados: init done 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f631c003900 con 0x7f632c136ba0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 --2- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 0x7f6318039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f631c021070 con 0x7f632c136ba0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 client.4314.objecter ms_dispatch 0x7f632c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 3 client.4314.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 3 client.4314.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 20 client.4314.objecter dump_active .. 0 homeless 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6330ce2700 1 --2- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 0x7f6318039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6330ce2700 10 monclient: get_auth_request con 0x7f6318036fb0 auth_method 0 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c0a4cc0 con 0x7f632c136ba0 2024-09-16T10:50:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6d3e655700 10 monclient: get_monmap_and_config 2024-09-16T10:50:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6d3e655700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6d3e655700 10 monclient: monmap: 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.973917+0000 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.973917+0000 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.037 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f6330ce2700 1 --2- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 0x7f6318039470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63200049b0 tx=0x7f632000a330 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f63297fa700 10 client.4314.objecter ms_handle_connect 0x7f6318036fb0 2024-09-16T10:50:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 client.?.objecter ms_handle_connect 0x7f9a88135c30 2024-09-16T10:50:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80004570 con 0x7f9a88135c30 2024-09-16T10:50:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 Processor -- start 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: dump: 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.041 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 5 monclient: authenticate success, global_id 4311 2024-09-16T10:50:05.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a800046f0 con 0x7f9a88135c30 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _renew_subs 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8813ada0 con 0x7f9a88135c30 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8001e430 con 0x7f9a88135c30 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a857fa700 10 monclient: dump: 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.043 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _renew_subs 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8811cfa0 con 0x7f9a88135c30 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 1 librados: init done 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a867fc700 4 set_mon_vals no callback set 2024-09-16T10:50:05.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 -- start start 2024-09-16T10:50:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient: init 2024-09-16T10:50:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T10:50:05.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 0x7f6d38131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-c con 0x7f6d380a10f0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6d3809b3e0 0x7f6d38131f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-a con 0x7f6d3809b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 0x7f6d3809e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-b con 0x7f6d38132480 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3809fc70 con 0x7f6d3809b3e0 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3809c0f0 con 0x7f6d380a10f0 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380a14d0 con 0x7f6d38132480 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3e655700 10 monclient(hunting): authenticate will time out at 863.553260s 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6d3809b3e0 0x7f6d38131f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6d3809b3e0 0x7f6d38131f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48918/0 (socket says 172.21.15.5:48918) 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 -- 172.21.15.5:0/3501271097 learned_addr learned my addr 172.21.15.5:0/3501271097 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): get_auth_request con 0x7f6d3809b3e0 auth_method 0 2024-09-16T10:50:05.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 0x7f6d3809e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): get_auth_request con 0x7f6d38132480 auth_method 0 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 1 --2- 172.21.15.5:0/3501271097 >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 0x7f6d38131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68005100 con 0x7f9a88135c30 2024-09-16T10:50:05.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 10 monclient(hunting): get_auth_request con 0x7f6d380a10f0 auth_method 0 2024-09-16T10:50:05.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 -- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 msgr2=0x7f6d3809e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 0x7f6d3809e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 -- 172.21.15.5:0/3501271097 >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 msgr2=0x7f6d38131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/3501271097 >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 0x7f6d38131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 monclient: found mon.noname-a 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 -- 172.21.15.5:0/3501271097 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3809be10 con 0x7f6d3809b3e0 2024-09-16T10:50:05.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3de54700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 0x7f6d3809e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:05.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3d653700 1 --2- 172.21.15.5:0/3501271097 >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 0x7f6d38131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:05.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d3809b3e0 0x7f6d38131f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6d380a4f00 tx=0x7f6d34009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213445fcd8ed15d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.972+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f631c015020 con 0x7f632c136ba0 2024-09-16T10:50:05.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f63297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f63297fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:05.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9a8001e590 con 0x7f9a88135c30 2024-09-16T10:50:05.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 1 --2- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 0x7f9a74039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f9a80049f90 con 0x7f9a88135c30 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 10 client.4311.objecter ms_dispatch 0x7f9a88000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 3 client.4311.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 3 client.4311.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 20 client.4311.objecter dump_active .. 0 homeless 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a8cf29700 1 --2- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 0x7f9a74039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a8cf29700 10 monclient: get_auth_request con 0x7f9a74036fb0 auth_method 0 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a857fa700 10 client.4311.objecter ms_handle_connect 0x7f9a74036fb0 2024-09-16T10:50:05.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.976+0000 7f9a8cf29700 1 --2- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 0x7f9a74039470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a780013e0 tx=0x7f9a78001370 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f57709d7700 10 monclient: get_monmap_and_config 2024-09-16T10:50:05.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f57709d7700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.968+0000 7f57709d7700 10 monclient: monmap: 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:04.975136+0000 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:04.975136+0000 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:05.056 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 Processor -- start 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 -- start start 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient: init 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f576c1326b0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 0x7f576c1378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-b con 0x7f576c1331a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 0x7f576c13a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-a con 0x7f576c137e30 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c09cfe0 con 0x7f576c1326b0 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c09ff00 con 0x7f576c137e30 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c09cd00 con 0x7f576c1331a0 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f57709d7700 10 monclient(hunting): authenticate will time out at 863.561155s 2024-09-16T10:50:05.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576b7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f576c1326b0 0x7f576c132a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59008/0 (socket says 172.21.15.5:59008) 2024-09-16T10:50:05.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576b7fe700 1 -- 172.21.15.5:0/1537340189 learned_addr learned my addr 172.21.15.5:0/1537340189 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:05.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a8001e840 con 0x7f9a88135c30 2024-09-16T10:50:05.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f9a857fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f9a857fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:05.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 --2- 172.21.15.5:0/1537340189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 0x7f576c1378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 1 --2- 172.21.15.5:0/1537340189 >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 0x7f576c13a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): get_auth_request con 0x7f576c137e30 auth_method 0 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): get_auth_request con 0x7f576c1331a0 auth_method 0 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 -- 172.21.15.5:0/1537340189 >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 msgr2=0x7f576c13a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 --2- 172.21.15.5:0/1537340189 >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 0x7f576c13a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 -- 172.21.15.5:0/1537340189 >> v1:172.21.15.5:6789/0 conn(0x7f576c1326b0 legacy=0x7f576c132a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 monclient: found mon.noname-b 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 -- 172.21.15.5:0/1537340189 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c09fc80 con 0x7f576c1331a0 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576bfff700 1 --2- 172.21.15.5:0/1537340189 >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 0x7f576c13a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.980+0000 7f576affd700 1 --2- 172.21.15.5:0/1537340189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 0x7f576c1378f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575c009ab0 tx=0x7f575c009dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=928995afc360b6bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 1 -- 172.21.15.5:0/1537340189 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00b850 con 0x7f576c1331a0 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: dump: 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.065 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 5 monclient: authenticate success, global_id 4268 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 1 -- 172.21.15.5:0/1537340189 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c00b9b0 con 0x7f576c1331a0 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 4 set_mon_vals no callback set 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 1 -- 172.21.15.5:0/1537340189 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00bd40 con 0x7f576c1331a0 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f576a7fc700 10 monclient: dump: 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.067 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 10 monclient: shutdown 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 -- 172.21.15.5:0/1537340189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 msgr2=0x7f576c1378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 --2- 172.21.15.5:0/1537340189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 0x7f576c1378f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f575c009ab0 tx=0x7f575c009dc0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 -- 172.21.15.5:0/1537340189 shutdown_connections 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 --2- 172.21.15.5:0/1537340189 >> v2:172.21.15.5:3300/0 conn(0x7f576c137e30 0x7f576c13a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 --2- 172.21.15.5:0/1537340189 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1331a0 0x7f576c1378f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.984+0000 7f57709d7700 1 -- 172.21.15.5:0/1537340189 >> 172.21.15.5:0/1537340189 conn(0x7f576c0a45d0 msgr2=0x7f576c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 1 -- 172.21.15.5:0/3501271097 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34011830 con 0x7f6d3809b3e0 2024-09-16T10:50:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: dump: 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 5 monclient: authenticate success, global_id 4317 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 1 -- 172.21.15.5:0/3501271097 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d34011990 con 0x7f6d3809b3e0 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 1 -- 172.21.15.5:0/3501271097 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34011d00 con 0x7f6d3809b3e0 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 4 set_mon_vals no callback set 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d2ffff700 10 monclient: dump: 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 10 monclient: shutdown 2024-09-16T10:50:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 -- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d3809b3e0 msgr2=0x7f6d38131f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d3809b3e0 0x7f6d38131f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6d380a4f00 tx=0x7f6d34009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 -- 172.21.15.5:0/3501271097 shutdown_connections 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d38132480 0x7f6d3809e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 --2- 172.21.15.5:0/3501271097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d3809b3e0 0x7f6d38131f40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 --2- 172.21.15.5:0/3501271097 >> v2:172.21.15.143:3300/0 conn(0x7f6d380a10f0 0x7f6d38131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 -- 172.21.15.5:0/3501271097 >> 172.21.15.5:0/3501271097 conn(0x7f6d380a45d0 msgr2=0x7f6d380a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 -- 172.21.15.5:0/3501271097 shutdown_connections 2024-09-16T10:50:05.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f57709d7700 1 -- 172.21.15.5:0/1537340189 shutdown_connections 2024-09-16T10:50:05.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f57709d7700 1 -- 172.21.15.5:0/1537340189 wait complete. 2024-09-16T10:50:05.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient: monmap: 2024-09-16T10:50:05.076 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:05.077 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:05.078 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:05.078 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:05.078 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.988+0000 7f6d3e655700 1 -- 172.21.15.5:0/3501271097 wait complete. 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: build_initial_monmap 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: monmap: 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:05.079 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 librados: starting msgr at 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 librados: starting objecter 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 Processor -- start 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 -- start start 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 librados: setting wanted keys 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 librados: calling monclient init 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: init 2024-09-16T10:50:05.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:05.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 0x7f6d381ca430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-c con 0x7f6d3809b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-a con 0x7f6d380a10f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 0x7f6d3811e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): picked mon.noname-b con 0x7f6d38132480 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d381cb410 con 0x7f6d380a10f0 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38070560 con 0x7f6d38132480 2024-09-16T10:50:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d38121820 con 0x7f6d3809b3e0 2024-09-16T10:50:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient(hunting): authenticate will time out at 863.573678s 2024-09-16T10:50:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 librados: starting msgr at 2024-09-16T10:50:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 librados: starting objecter 2024-09-16T10:50:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 Processor -- start 2024-09-16T10:50:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 -- start start 2024-09-16T10:50:05.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 librados: setting wanted keys 2024-09-16T10:50:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 librados: calling monclient init 2024-09-16T10:50:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient: init 2024-09-16T10:50:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3d653700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 0x7f6d381ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1326b0 0x7f576c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-c con 0x7f576c1326b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576c1331a0 0x7f576c06d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-b con 0x7f576c1331a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): picked mon.noname-a con 0x7f576c137e30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c070e60 con 0x7f576c137e30 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0a26f0 con 0x7f576c1331a0 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c11e940 con 0x7f576c1326b0 2024-09-16T10:50:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:05.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48942/0 (socket says 172.21.15.5:48942) 2024-09-16T10:50:05.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 -- 172.21.15.5:0/2734989160 learned_addr learned my addr 172.21.15.5:0/2734989160 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:05.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): get_auth_request con 0x7f6d380a10f0 auth_method 0 2024-09-16T10:50:05.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f57709d7700 10 monclient(hunting): authenticate will time out at 863.574051s 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3de54700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 0x7f6d3811e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-16T10:50:05.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 -- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 msgr2=0x7f6d381ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48958/0 (socket says 172.21.15.5:48958) 2024-09-16T10:50:05.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 -- 172.21.15.5:0/2741617339 learned_addr learned my addr 172.21.15.5:0/2741617339 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:05.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): get_auth_request con 0x7f576c137e30 auth_method 0 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576c1331a0 0x7f576c06d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): get_auth_request con 0x7f576c1331a0 auth_method 0 2024-09-16T10:50:05.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576b7fe700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1326b0 0x7f576c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576b7fe700 10 monclient(hunting): get_auth_request con 0x7f576c1326b0 auth_method 0 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:05.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 -- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1326b0 msgr2=0x7f576c06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1326b0 0x7f576c06d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 -- 172.21.15.5:0/2741617339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576c1331a0 msgr2=0x7f576c06d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576c1331a0 0x7f576c06d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 monclient: found mon.noname-a 2024-09-16T10:50:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 -- 172.21.15.5:0/2741617339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764009a70 con 0x7f576c137e30 2024-09-16T10:50:05.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 0x7f6d381ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 -- 172.21.15.5:0/2734989160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 msgr2=0x7f6d3811e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 0x7f6d3811e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient: _finish_hunting 0 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 monclient: found mon.noname-a 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 -- 172.21.15.5:0/2734989160 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d28008050 con 0x7f6d380a10f0 2024-09-16T10:50:05.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3de54700 10 monclient: get_auth_request con 0x7f6d38132480 auth_method 0 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3de54700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 0x7f6d3811e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3d653700 10 monclient: get_auth_request con 0x7f6d3809b3e0 auth_method 0 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3d653700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 0x7f6d381ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3ce52700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6d34014040 tx=0x7f6d34004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 client.?.objecter ms_handle_connect 0x7f6d380a10f0 2024-09-16T10:50:05.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f576bfff700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f576400efb0 tx=0x7f576400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f5753fff700 10 client.?.objecter ms_handle_connect 0x7f576c137e30 2024-09-16T10:50:05.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f5753fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:05.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34004510 con 0x7f6d380a10f0 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: dump: 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.103 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d340036c0 con 0x7f6d380a10f0 2024-09-16T10:50:05.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 5 monclient: authenticate success, global_id 4326 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: _renew_subs 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34003a70 con 0x7f6d380a10f0 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d34009570 con 0x7f6d380a10f0 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 10 monclient: dump: 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.105 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3e655700 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3e655700 10 monclient: _renew_subs 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3e655700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d38121f10 con 0x7f6d380a10f0 2024-09-16T10:50:05.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3e655700 1 librados: init done 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.992+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400cd40 con 0x7f576c137e30 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: dump: 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.107 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576400f040 con 0x7f576c137e30 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400cea0 con 0x7f576c137e30 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:05.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 monclient: dump: 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:05.109 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:05.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6d34021020 con 0x7f6d380a10f0 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 5 monclient: authenticate success, global_id 4329 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 10 monclient: _renew_subs 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0094b0 con 0x7f576c137e30 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 10 monclient: _renew_subs 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c09cad0 con 0x7f576c137e30 2024-09-16T10:50:05.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f57709d7700 1 librados: init done 2024-09-16T10:50:05.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 1 --2- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 0x7f6d1c039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f6d34048950 con 0x7f6d380a10f0 2024-09-16T10:50:05.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 10 client.4326.objecter ms_dispatch 0x7f6d38000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:05.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 3 client.4326.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:05.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 3 client.4326.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 20 client.4326.objecter dump_active .. 0 homeless 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3d653700 1 --2- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 0x7f6d1c039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3d653700 10 monclient: get_auth_request con 0x7f6d1c036fb0 auth_method 0 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f57640146a0 con 0x7f576c137e30 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d2dffb700 10 client.4326.objecter ms_handle_connect 0x7f6d1c036fb0 2024-09-16T10:50:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f6d3d653700 1 --2- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 0x7f6d1c039470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009430 tx=0x7f6d30009600 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 1 --2- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 0x7f57540395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f57640484e0 con 0x7f576c137e30 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 10 client.4329.objecter ms_dispatch 0x7f576c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 3 client.4329.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 3 client.4329.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:05.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f5753fff700 20 client.4329.objecter dump_active .. 0 homeless 2024-09-16T10:50:05.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f576b7fe700 1 --2- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 0x7f57540395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:05.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:04.996+0000 7f576b7fe700 10 monclient: get_auth_request con 0x7f5754037110 auth_method 0 2024-09-16T10:50:05.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.000+0000 7f6d2effd700 4 set_mon_vals no callback set 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.000+0000 7f5769ffb700 4 set_mon_vals no callback set 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.000+0000 7f5753fff700 10 client.4329.objecter ms_handle_connect 0x7f5754037110 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.000+0000 7f576b7fe700 1 --2- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 0x7f57540395d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5760000f60 tx=0x7f5760009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f57709d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f57709d7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f57709d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c005100 con 0x7f576c137e30 2024-09-16T10:50:05.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f6d3e655700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f6d3e655700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f6d3e655700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.004+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d18005100 con 0x7f6d380a10f0 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.008+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f57640076f0 con 0x7f576c137e30 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.008+0000 7f5753fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.008+0000 7f5753fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.020+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6d34015080 con 0x7f6d380a10f0 2024-09-16T10:50:05.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.020+0000 7f6d2dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:05.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.020+0000 7f6d2dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f3887fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f3887fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f3887fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3888070eb0 con 0x7f38880713f0 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f386ffff700 1 -- 172.21.15.5:0/1745027948 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f387c02b090 con 0x7f38880713f0 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f386ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:05.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.128+0000 7f386ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.139 INFO:teuthology.orchestra.run.smithi005.stdout:47244640257 2024-09-16T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 10 librados: watch_flush enter 2024-09-16T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 10 librados: watch_flush exit 2024-09-16T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 20 client.4247.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 10 client.4247.objecter successfully canceled tick 2024-09-16T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 msgr2=0x7f3874041c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 0x7f3874041c30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3878000c00 tx=0x7f3878008030 comp rx=0 tx=0).stop 2024-09-16T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 10 monclient: shutdown 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 msgr2=0x7f38881c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f387c00e790 tx=0x7f387c00eaa0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 shutdown_connections 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38881ca510 0x7f38881ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f38880713f0 0x7f38881c4560 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> v2:172.21.15.143:6812/1098205034 conn(0x7f387403f770 0x7f3874041c30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 --2- 172.21.15.5:0/1745027948 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3888070ad0 0x7f38881c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 >> 172.21.15.5:0/1745027948 conn(0x7f38880a3200 msgr2=0x7f38880a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 shutdown_connections 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 -- 172.21.15.5:0/1745027948 wait complete. 2024-09-16T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.132+0000 7f3887fff700 1 librados: shutdown 2024-09-16T10:50:05.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.4 2024-09-16T10:50:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f57709d7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f57709d7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f57709d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f574c0053b0 con 0x7f576c137e30 2024-09-16T10:50:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f5753fff700 1 -- 172.21.15.5:0/2741617339 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5764007330 con 0x7f576c137e30 2024-09-16T10:50:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f5753fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:05.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.180+0000 7f5753fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.191 INFO:teuthology.orchestra.run.smithi005.stdout:47244640257 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 10 librados: watch_flush enter 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 10 librados: watch_flush exit 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 20 client.4329.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 10 client.4329.objecter successfully canceled tick 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 msgr2=0x7f57540395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 0x7f57540395d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5760000f60 tx=0x7f5760009040 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 10 monclient: shutdown 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 msgr2=0x7f576c06de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f576400efb0 tx=0x7f576400c3a0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 shutdown_connections 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5754037110 0x7f57540395d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f576c137e30 0x7f576c06de10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576c1331a0 0x7f576c06d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 --2- 172.21.15.5:0/2741617339 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f576c1326b0 0x7f576c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 >> 172.21.15.5:0/2741617339 conn(0x7f576c0a45d0 msgr2=0x7f576c12f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 shutdown_connections 2024-09-16T10:50:05.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 -- 172.21.15.5:0/2741617339 wait complete. 2024-09-16T10:50:05.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.184+0000 7f57709d7700 1 librados: shutdown 2024-09-16T10:50:05.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.2 2024-09-16T10:50:05.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f544f5cc700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:05.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f544f5cc700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:05.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f544f5cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:05.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5448072360 con 0x7f5448130f30 2024-09-16T10:50:05.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f543dffb700 1 -- 172.21.15.5:0/1903236750 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f544003b090 con 0x7f5448130f30 2024-09-16T10:50:05.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f543dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:05.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.252+0000 7f543dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.262 INFO:teuthology.orchestra.run.smithi005.stdout:47244640257 2024-09-16T10:50:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 10 librados: watch_flush enter 2024-09-16T10:50:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 10 librados: watch_flush exit 2024-09-16T10:50:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 20 client.4250.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 10 client.4250.objecter successfully canceled tick 2024-09-16T10:50:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 msgr2=0x7f542c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 0x7f542c039420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5444001a80 tx=0x7f5444009040 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 10 monclient: shutdown 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 msgr2=0x7f544811a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5440002230 tx=0x7f544000cf50 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 shutdown_connections 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5448130f30 0x7f544811a680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> v2:172.21.15.143:6812/1098205034 conn(0x7f542c036f60 0x7f542c039420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f544813b200 0x7f54481309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 --2- 172.21.15.5:0/1903236750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5448135f10 0x7f5448139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 >> 172.21.15.5:0/1903236750 conn(0x7f54480a0b30 msgr2=0x7f54480a32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 shutdown_connections 2024-09-16T10:50:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 -- 172.21.15.5:0/1903236750 wait complete. 2024-09-16T10:50:05.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f544f5cc700 1 librados: shutdown 2024-09-16T10:50:05.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f9a8df2b700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:05.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f9a8df2b700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f9a8df2b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.256+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9a68005680 con 0x7f9a88135c30 2024-09-16T10:50:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a857fa700 1 -- 172.21.15.5:0/1444913320 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a80022020 con 0x7f9a88135c30 2024-09-16T10:50:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a857fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a857fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.268 INFO:teuthology.orchestra.run.smithi005.stdout:47244640257 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 10 librados: watch_flush enter 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 10 librados: watch_flush exit 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 20 client.4311.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 10 client.4311.objecter successfully canceled tick 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 msgr2=0x7f9a74039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 0x7f9a74039470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9a780013e0 tx=0x7f9a78001370 comp rx=0 tx=0).stop 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 10 monclient: shutdown 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 msgr2=0x7f9a88134970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000b370 tx=0x7f9a80004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 shutdown_connections 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a8813afb0 0x7f9a8811b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a74036fb0 0x7f9a74039470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a88135c30 0x7f9a88134970 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 --2- 172.21.15.5:0/1444913320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a880a04f0 0x7f9a881ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 >> 172.21.15.5:0/1444913320 conn(0x7f9a880a45d0 msgr2=0x7f9a8809c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 shutdown_connections 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 -- 172.21.15.5:0/1444913320 wait complete. 2024-09-16T10:50:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.260+0000 7f9a8df2b700 1 librados: shutdown 2024-09-16T10:50:05.277 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.3 2024-09-16T10:50:05.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2024-09-16T10:50:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f6331ce4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f6331ce4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f6331ce4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f632c09b0d0 con 0x7f632c136ba0 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f63297fa700 1 -- 172.21.15.5:0/1116555497 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f631c033070 con 0x7f632c136ba0 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f63297fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.292+0000 7f63297fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.302 INFO:teuthology.orchestra.run.smithi005.stdout:47244640257 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 10 librados: watch_flush enter 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 10 librados: watch_flush exit 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 20 client.4314.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 10 client.4314.objecter successfully canceled tick 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 msgr2=0x7f6318039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 0x7f6318039470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63200049b0 tx=0x7f632000a330 comp rx=0 tx=0).stop 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 10 monclient: shutdown 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 msgr2=0x7f632c12df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 0x7f632c12df80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f631c016000 tx=0x7f631c004770 comp rx=0 tx=0).stop 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 shutdown_connections 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6318036fb0 0x7f6318039470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f632c136ba0 0x7f632c12df80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f632c0a10f0 0x7f632c12da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 --2- 172.21.15.5:0/1116555497 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f632c09b3e0 0x7f632c1c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 >> 172.21.15.5:0/1116555497 conn(0x7f632c0a45d0 msgr2=0x7f632c12e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 shutdown_connections 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 -- 172.21.15.5:0/1116555497 wait complete. 2024-09-16T10:50:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.296+0000 7f6331ce4700 1 librados: shutdown 2024-09-16T10:50:05.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.1 2024-09-16T10:50:05.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:05.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:05.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:05.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6d18005680 con 0x7f6d380a10f0 2024-09-16T10:50:05.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d2dffb700 1 -- 172.21.15.5:0/2734989160 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6d34029030 con 0x7f6d380a10f0 2024-09-16T10:50:05.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d2dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:05.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d2dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:05.336 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 librados: watch_flush enter 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 librados: watch_flush exit 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 20 client.4326.objecter shutdown clearing up homeless session... 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 client.4326.objecter successfully canceled tick 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 msgr2=0x7f6d1c039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 0x7f6d1c039470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009430 tx=0x7f6d30009600 comp rx=0 tx=0).stop 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 10 monclient: shutdown 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:05.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 msgr2=0x7f6d38130db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6d34014040 tx=0x7f6d34004770 comp rx=0 tx=0).stop 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 shutdown_connections 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d38132480 0x7f6d3811e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d1c036fb0 0x7f6d1c039470 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d380a10f0 0x7f6d38130db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 --2- 172.21.15.5:0/2734989160 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d3809b3e0 0x7f6d381ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 >> 172.21.15.5:0/2734989160 conn(0x7f6d380a45d0 msgr2=0x7f6d381b9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 shutdown_connections 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 -- 172.21.15.5:0/2734989160 wait complete. 2024-09-16T10:50:05.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:05.328+0000 7f6d3e655700 1 librados: shutdown 2024-09-16T10:50:05.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.5 2024-09-16T10:50:06.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T10:50:06.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:50:06.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.264+0000 7f501b2cf700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.264+0000 7f501b2cf700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.264+0000 7f501b2cf700 10 monclient: monmap: 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.272583+0000 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.272583+0000 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.275 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.276 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.276 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.276 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 Processor -- start 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 -- start start 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient: init 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50140a19f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50140a24e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f501413c2e0 0x7f501413e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-c con 0x7f501413c2e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501409e0b0 con 0x7f50140a19f0 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501409f9f0 con 0x7f50140a24e0 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501409e2f0 con 0x7f501413c2e0 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient(hunting): authenticate will time out at 864.849565s 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f50140a19f0 0x7f50140a1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59018/0 (socket says 172.21.15.5:59018) 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 -- 172.21.15.5:0/1291808989 learned_addr learned my addr 172.21.15.5:0/1291808989 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 1 --2- 172.21.15.5:0/1291808989 >> v2:172.21.15.143:3300/0 conn(0x7f501413c2e0 0x7f501413e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753961808 0 0) 0x7f501409e0b0 con 0x7f50140a19f0 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): _negotiate 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): my global_id is 4332 2024-09-16T10:50:06.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc0035d0 con 0x7f50140a19f0 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): get_auth_request con 0x7f501413c2e0 auth_method 0 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644160379 0 0) 0x7f4ffc0035d0 con 0x7f50140a19f0 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 >> v2:172.21.15.143:3300/0 conn(0x7f501413c2e0 msgr2=0x7f501413e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 --2- 172.21.15.5:0/1291808989 >> v2:172.21.15.143:3300/0 conn(0x7f501413c2e0 0x7f501413e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 >> v1:172.21.15.143:6789/0 conn(0x7f50140a24e0 legacy=0x7f501413abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:50:06.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501409f150 con 0x7f50140a19f0 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f500c0019f0 con 0x7f50140a19f0 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 5 monclient: authenticate success, global_id 4332 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: dump: 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.283 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 >> v1:172.21.15.5:6789/0 conn(0x7f50140a19f0 legacy=0x7f50140a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:06.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 0x7f4ffc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): picked mon.c con 0x7f4ffc003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ffc004870 0x7f4ffc004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): picked mon.a con 0x7f4ffc004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc005430 0x7f4ffc00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): picked mon.b con 0x7f4ffc005430 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc00c290 con 0x7f4ffc004870 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc00c400 con 0x7f4ffc005430 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc00c5c0 con 0x7f4ffc003a20 2024-09-16T10:50:06.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 0x7f4ffc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): get_auth_request con 0x7f4ffc003a20 auth_method 0 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc005430 0x7f4ffc00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): get_auth_request con 0x7f4ffc005430 auth_method 0 2024-09-16T10:50:06.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f5019acc700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ffc004870 0x7f4ffc004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient(hunting): handle_auth_done global_id 4332 payload 293 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 -- 172.21.15.5:0/1291808989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc005430 msgr2=0x7f4ffc00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc005430 0x7f4ffc00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 -- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ffc004870 msgr2=0x7f4ffc004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ffc004870 0x7f4ffc004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 monclient: found mon.c 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 -- 172.21.15.5:0/1291808989 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0035d0 con 0x7f4ffc003a20 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f5019acc700 10 monclient: get_auth_request con 0x7f4ffc004870 auth_method 0 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501aace700 1 -- 172.21.15.5:0/1291808989 reap_dead start 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f5019acc700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ffc004870 0x7f4ffc004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501a2cd700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 0x7f4ffc003e40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f500c00a830 tx=0x7f4ffc0058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=880e6eb1b6d32ae9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c00a2f0 con 0x7f4ffc003a20 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: dump: 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c0072f0 con 0x7f4ffc003a20 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 1 -- 172.21.15.5:0/1291808989 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c0076a0 con 0x7f4ffc003a20 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f501b2cf700 4 set_mon_vals no callback set 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.268+0000 7f50192cb700 10 monclient: dump: 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.290 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: shutdown 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 msgr2=0x7f4ffc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 0x7f4ffc003e40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f500c00a830 tx=0x7f4ffc0058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- 172.21.15.5:0/1291808989 shutdown_connections 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 --2- 172.21.15.5:0/1291808989 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ffc003a20 0x7f4ffc003e40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- 172.21.15.5:0/1291808989 >> 172.21.15.5:0/1291808989 conn(0x7f50140a3200 msgr2=0x7f5014073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- 172.21.15.5:0/1291808989 shutdown_connections 2024-09-16T10:50:06.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- 172.21.15.5:0/1291808989 wait complete. 2024-09-16T10:50:06.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T10:50:06.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: monmap: 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.298 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 librados: starting msgr at 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 librados: starting objecter 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 Processor -- start 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- start start 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 librados: setting wanted keys 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 librados: calling monclient init 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: init 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc003a20 0x7f5014137960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-b con 0x7f4ffc003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5014137ea0 0x7f5014120a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-a con 0x7f5014137ea0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): picked mon.noname-c con 0x7f5014138280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014122ff0 con 0x7f5014137ea0 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140a5650 con 0x7f4ffc003a20 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501409d1d0 con 0x7f5014138280 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501b2cf700 10 monclient(hunting): authenticate will time out at 864.855277s 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38042/0 (socket says 172.21.15.5:38042) 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 -- 172.21.15.5:0/3799565645 learned_addr learned my addr 172.21.15.5:0/3799565645 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5014137ea0 0x7f5014120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): get_auth_request con 0x7f5014138280 auth_method 0 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc003a20 0x7f5014137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): get_auth_request con 0x7f5014137ea0 auth_method 0 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): get_auth_request con 0x7f4ffc003a20 auth_method 0 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f5019acc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501a2cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 -- 172.21.15.5:0/3799565645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc003a20 msgr2=0x7f5014137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc003a20 0x7f5014137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 -- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5014137ea0 msgr2=0x7f5014120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5014137ea0 0x7f5014120a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.272+0000 7f501aace700 1 -- 172.21.15.5:0/3799565645 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014122d10 con 0x7f5014138280 2024-09-16T10:50:06.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 client.?.objecter ms_handle_connect 0x7f5014138280 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501aace700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f501409e5d0 tx=0x7f5004010810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004004440 con 0x7f5014138280 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: dump: 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.304 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500400c050 con 0x7f5014138280 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 5 monclient: authenticate success, global_id 4274 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _renew_subs 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5014121e20 con 0x7f5014138280 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004003900 con 0x7f5014138280 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5013fff700 4 set_mon_vals no callback set 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 monclient: dump: 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.306 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _renew_subs 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50141389b0 con 0x7f5014138280 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 1 librados: init done 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5004003a60 con 0x7f5014138280 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 --2- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 0x7f5000039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f5004047360 con 0x7f5014138280 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 client.4274.objecter ms_dispatch 0x7f5014000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 3 client.4274.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 3 client.4274.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 20 client.4274.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5014066830 con 0x7f5014138280 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501a2cd700 1 --2- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 0x7f5000039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501a2cd700 10 monclient: get_auth_request con 0x7f5000036f60 auth_method 0 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f5012ffd700 10 client.4274.objecter ms_handle_connect 0x7f5000036f60 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.276+0000 7f501a2cd700 1 --2- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 0x7f5000039420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f500c016040 tx=0x7f500c007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.284+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5004018020 con 0x7f5014138280 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.284+0000 7f5012ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.284+0000 7f5012ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:50:06.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: monmap: 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.322677+0000 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.322677+0000 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.325 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.326 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.326 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 Processor -- start 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 -- start start 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: init 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 0x7f5bc00a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc00a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bc009f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5bc00a3510 0x7f5bc01360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-c con 0x7f5bc00a3510 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc00a3c40 con 0x7f5bc009f780 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc00a4300 con 0x7f5bc00a3510 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc009fda0 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc6ed2700 10 monclient(hunting): authenticate will time out at 864.899244s 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc56cf700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f5bc009f780 0x7f5bc01359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59034/0 (socket says 172.21.15.5:59034) 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc56cf700 1 -- 172.21.15.5:0/1995426887 learned_addr learned my addr 172.21.15.5:0/1995426887 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/1995426887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 0x7f5bc00a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): get_auth_request con 0x7f5bc00a2bf0 auth_method 0 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc66d1700 1 --2- 172.21.15.5:0/1995426887 >> v2:172.21.15.143:3300/0 conn(0x7f5bc00a3510 0x7f5bc01360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc66d1700 10 monclient(hunting): get_auth_request con 0x7f5bc00a3510 auth_method 0 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc66d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc66d1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc66d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 -- 172.21.15.5:0/1995426887 >> v2:172.21.15.143:3300/0 conn(0x7f5bc00a3510 msgr2=0x7f5bc01360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/1995426887 >> v2:172.21.15.143:3300/0 conn(0x7f5bc00a3510 0x7f5bc01360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 -- 172.21.15.5:0/1995426887 >> v1:172.21.15.5:6789/0 conn(0x7f5bc009f780 legacy=0x7f5bc01359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 monclient: found mon.noname-b 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.316+0000 7f5bc5ed0700 1 -- 172.21.15.5:0/1995426887 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc00a0080 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/1995426887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 0x7f5bc00a2fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009850 tx=0x7f5bb0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02966fcfbaeb28e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 1 -- 172.21.15.5:0/1995426887 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0011830 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: dump: 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.337 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 1 -- 172.21.15.5:0/1995426887 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb0011990 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 1 -- 172.21.15.5:0/1995426887 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0011d20 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 5 monclient: authenticate success, global_id 4277 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 4 set_mon_vals no callback set 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc4ece700 10 monclient: dump: 2024-09-16T10:50:06.338 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 10 monclient: shutdown 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/1995426887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 msgr2=0x7f5bc00a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/1995426887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 0x7f5bc00a2fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009850 tx=0x7f5bb0009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/1995426887 shutdown_connections 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/1995426887 >> v2:172.21.15.143:3300/0 conn(0x7f5bc00a3510 0x7f5bc01360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/1995426887 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a2bf0 0x7f5bc00a2fd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/1995426887 >> 172.21.15.5:0/1995426887 conn(0x7f5bc00a53e0 msgr2=0x7f5bc0073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/1995426887 shutdown_connections 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/1995426887 wait complete. 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 10 monclient: monmap: 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.340 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 librados: starting msgr at 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 librados: starting objecter 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 Processor -- start 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 -- start start 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 librados: setting wanted keys 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 1 librados: calling monclient init 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.320+0000 7f5bc6ed2700 10 monclient: init 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-a con 0x7f5bc009f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 0x7f5bc0136010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc00a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): picked mon.noname-c con 0x7f5bc00a3510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0136a90 con 0x7f5bc009f780 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc01cb500 con 0x7f5bc00a2bf0 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00a7990 con 0x7f5bc00a3510 2024-09-16T10:50:06.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient(hunting): authenticate will time out at 864.904966s 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38060/0 (socket says 172.21.15.5:38060) 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48966/0 (socket says 172.21.15.5:48966) 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 -- 172.21.15.5:0/3687835857 learned_addr learned my addr 172.21.15.5:0/3687835857 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): get_auth_request con 0x7f5bc00a3510 auth_method 0 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): get_auth_request con 0x7f5bc009f780 auth_method 0 2024-09-16T10:50:06.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 0x7f5bc0136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 10 monclient(hunting): get_auth_request con 0x7f5bc00a2bf0 auth_method 0 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 -- 172.21.15.5:0/3687835857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 msgr2=0x7f5bc0136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 0x7f5bc0136010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 -- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 msgr2=0x7f5bc01ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 -- 172.21.15.5:0/3687835857 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb00094b0 con 0x7f5bc00a3510 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:06.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc56cf700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 0x7f5bc0136010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc66d1700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8007ea0 tx=0x7f5bb800e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 client.?.objecter ms_handle_connect 0x7f5bc00a3510 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800eea0 con 0x7f5bc00a3510 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: dump: 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.346 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb8014420 con 0x7f5bc00a3510 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb80147d0 con 0x7f5bc00a3510 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb67fc700 10 monclient: dump: 2024-09-16T10:50:06.347 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 5 monclient: authenticate success, global_id 4280 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _renew_subs 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bb77fe700 4 set_mon_vals no callback set 2024-09-16T10:50:06.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc009f4e0 con 0x7f5bc00a3510 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _renew_subs 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc0121d90 con 0x7f5bc00a3510 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.324+0000 7f5bc6ed2700 1 librados: init done 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5bb8014930 con 0x7f5bc00a3510 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc6ed2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 1 --2- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 0x7f5ba4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f5bb8047f40 con 0x7f5bc00a3510 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 10 client.4280.objecter ms_dispatch 0x7f5bc0000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 3 client.4280.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 3 client.4280.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 20 client.4280.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc6ed2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc6ed2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc00a2fd0 con 0x7f5bc00a3510 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 0x7f5ba4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc5ed0700 10 monclient: get_auth_request con 0x7f5ba4036f60 auth_method 0 2024-09-16T10:50:06.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bc5ed0700 1 --2- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 0x7f5ba4039420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000c00 tx=0x7f5bb0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.328+0000 7f5bb67fc700 10 client.4280.objecter ms_handle_connect 0x7f5ba4036f60 2024-09-16T10:50:06.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.336+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5bb80122e0 con 0x7f5bc00a3510 2024-09-16T10:50:06.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.336+0000 7f5bb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.336+0000 7f5bb67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T10:50:06.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.400+0000 7fb981f28700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.400+0000 7fb981f28700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.400+0000 7fb981f28700 10 monclient: monmap: 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.408198+0000 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.408198+0000 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.411 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.412 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 Processor -- start 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- start start 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient: init 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb97c0a2bf0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 0x7fb97c13ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-b con 0x7fb97c09f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb97c0a36e0 0x7fb97c13d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-a con 0x7fb97c0a36e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c0a3c40 con 0x7fb97c0a2bf0 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c0a4300 con 0x7fb97c0a36e0 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c09fda0 con 0x7fb97c09f780 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient(hunting): authenticate will time out at 864.985026s 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb97c0a36e0 0x7fb97c13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb97c0a36e0 0x7fb97c13d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48970/0 (socket says 172.21.15.5:48970) 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 -- 172.21.15.5:0/2963471363 learned_addr learned my addr 172.21.15.5:0/2963471363 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): get_auth_request con 0x7fb97c0a36e0 auth_method 0 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 0x7fb97c13ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): get_auth_request con 0x7fb97c09f780 auth_method 0 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 -- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 msgr2=0x7fb97c13ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 0x7fb97c13ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 -- 172.21.15.5:0/2963471363 >> v1:172.21.15.5:6789/0 conn(0x7fb97c0a2bf0 legacy=0x7fb97c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 monclient: found mon.noname-a 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 -- 172.21.15.5:0/2963471363 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0a0080 con 0x7fb97c0a36e0 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97bfff700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 0x7fb97c13ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981727700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c0a36e0 0x7fb97c13d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb974009e50 tx=0x7fb97400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1cb093f8aa29aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 1 -- 172.21.15.5:0/2963471363 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97400e070 con 0x7fb97c0a36e0 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: dump: 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.419 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 1 -- 172.21.15.5:0/2963471363 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97400b910 con 0x7fb97c0a36e0 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 1 -- 172.21.15.5:0/2963471363 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97400bca0 con 0x7fb97c0a36e0 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb97b7fe700 10 monclient: dump: 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.420 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 5 monclient: authenticate success, global_id 4344 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 4 set_mon_vals no callback set 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 10 monclient: shutdown 2024-09-16T10:50:06.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c0a36e0 msgr2=0x7fb97c13d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c0a36e0 0x7fb97c13d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb974009e50 tx=0x7fb97400b330 comp rx=0 tx=0).stop 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.404+0000 7fb981f28700 1 -- 172.21.15.5:0/2963471363 shutdown_connections 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c0a36e0 0x7fb97c13d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 --2- 172.21.15.5:0/2963471363 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c09f780 0x7fb97c13ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- 172.21.15.5:0/2963471363 >> 172.21.15.5:0/2963471363 conn(0x7fb97c0a53e0 msgr2=0x7fb97c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- 172.21.15.5:0/2963471363 shutdown_connections 2024-09-16T10:50:06.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- 172.21.15.5:0/2963471363 wait complete. 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient: monmap: 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.426 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 librados: starting msgr at 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 librados: starting objecter 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 Processor -- start 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- start start 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 librados: setting wanted keys 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 librados: calling monclient init 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient: init 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 0x7fb97c135410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-a con 0x7fb97c09f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb97c0a2bf0 0x7fb97c135950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-b con 0x7fb97c0a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 0x7fb97c13aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): picked mon.noname-c con 0x7fb97c0a36e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c11fb30 con 0x7fb97c09f780 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c13d4f0 con 0x7fb97c0a2bf0 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c09f4e0 con 0x7fb97c0a36e0 2024-09-16T10:50:06.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981f28700 10 monclient(hunting): authenticate will time out at 864.990288s 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 0x7fb97c135410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 0x7fb97c13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 0x7fb97c135410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48986/0 (socket says 172.21.15.5:48986) 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 1 -- 172.21.15.5:0/481202021 learned_addr learned my addr 172.21.15.5:0/481202021 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): get_auth_request con 0x7fb97c0a36e0 auth_method 0 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb97c0a2bf0 0x7fb97c135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): get_auth_request con 0x7fb97c09f780 auth_method 0 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): get_auth_request con 0x7fb97c0a2bf0 auth_method 0 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb97bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb980f26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 -- 172.21.15.5:0/481202021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb97c0a2bf0 msgr2=0x7fb97c135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb97c0a2bf0 0x7fb97c135950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 -- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 msgr2=0x7fb97c135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 0x7fb97c135410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 -- 172.21.15.5:0/481202021 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c0094b0 con 0x7fb97c0a36e0 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb981727700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 0x7fb97c13aa20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb97c074960 tx=0x7fb974004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 10 client.?.objecter ms_handle_connect 0x7fb97c0a36e0 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740047c0 con 0x7fb97c0a36e0 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.408+0000 7fb9797fa700 10 monclient: dump: 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.432 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 5 monclient: authenticate success, global_id 4286 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 10 monclient: _renew_subs 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb974009b30 con 0x7fb97c0a36e0 2024-09-16T10:50:06.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 10 monclient: _renew_subs 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c11ef80 con 0x7fb97c0a36e0 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb981f28700 1 librados: init done 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb974004920 con 0x7fb97c0a36e0 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97401c660 con 0x7fb97c0a36e0 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 monclient: dump: 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.434 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb97affd700 4 set_mon_vals no callback set 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb97401c800 con 0x7fb97c0a36e0 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 1 --2- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 0x7fb964039560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7fb974048f20 con 0x7fb97c0a36e0 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 client.4286.objecter ms_dispatch 0x7fb97c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 3 client.4286.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 3 client.4286.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 20 client.4286.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb980f26700 1 --2- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 0x7fb964039560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb980f26700 10 monclient: get_auth_request con 0x7fb9640370a0 auth_method 0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb9797fa700 10 client.4286.objecter ms_handle_connect 0x7fb9640370a0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.412+0000 7fb980f26700 1 --2- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 0x7fb964039560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb970001940 tx=0x7fb970009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.416+0000 7fb981f28700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.416+0000 7fb981f28700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.416+0000 7fb981f28700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.416+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c005100 con 0x7fb97c0a36e0 2024-09-16T10:50:06.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 10 monclient: monmap: 2024-09-16T10:50:06.438 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.427158+0000 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.427158+0000 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 1 Processor -- start 2024-09-16T10:50:06.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 1 -- start start 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.420+0000 7f8e68b69700 10 monclient: init 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e641336c0 0x7f8e64133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-c con 0x7f8e641336c0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e6409d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e6409db20 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6409df60 con 0x7f8e6409db20 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e640a0260 con 0x7f8e6409d6a0 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6409fef0 con 0x7f8e641336c0 2024-09-16T10:50:06.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e68b69700 10 monclient(hunting): authenticate will time out at 865.004433s 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e63fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8e6409db20 0x7f8e6409eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59048/0 (socket says 172.21.15.5:59048) 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e63fff700 1 -- 172.21.15.5:0/1595361432 learned_addr learned my addr 172.21.15.5:0/1595361432 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 --2- 172.21.15.5:0/1595361432 >> v2:172.21.15.143:3300/0 conn(0x7f8e641336c0 0x7f8e64133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): get_auth_request con 0x7f8e641336c0 auth_method 0 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb974024770 con 0x7fb97c0a36e0 2024-09-16T10:50:06.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7fb9797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7fb9797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 1 -- 172.21.15.5:0/1595361432 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181350827 0 0) 0x7f8e6409df60 con 0x7f8e6409db20 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 10 monclient(hunting): _negotiate 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 10 monclient(hunting): my global_id is 4353 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e627fc700 1 -- 172.21.15.5:0/1595361432 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c0035d0 con 0x7f8e6409db20 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 -- 172.21.15.5:0/1595361432 >> v1:172.21.15.143:6789/0 conn(0x7f8e6409d6a0 legacy=0x7f8e6409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 -- 172.21.15.5:0/1595361432 >> v1:172.21.15.5:6789/0 conn(0x7f8e6409db20 legacy=0x7f8e6409eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 -- 172.21.15.5:0/1595361432 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640a1360 con 0x7f8e641336c0 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.424+0000 7f8e637fe700 1 --2- 172.21.15.5:0/1595361432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e641336c0 0x7f8e64133aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009850 tx=0x7f8e58009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d800b658cd9e3fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 1 -- 172.21.15.5:0/1595361432 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58011830 con 0x7f8e641336c0 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: dump: 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.446 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 5 monclient: authenticate success, global_id 4339 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 1 -- 172.21.15.5:0/1595361432 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58011990 con 0x7f8e641336c0 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 4 set_mon_vals no callback set 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 1 -- 172.21.15.5:0/1595361432 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58011d40 con 0x7f8e641336c0 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e627fc700 10 monclient: dump: 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.448 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 10 monclient: shutdown 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 1 -- 172.21.15.5:0/1595361432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e641336c0 msgr2=0x7f8e64133aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 1 --2- 172.21.15.5:0/1595361432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e641336c0 0x7f8e64133aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009850 tx=0x7f8e58009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 1 -- 172.21.15.5:0/1595361432 shutdown_connections 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 1 --2- 172.21.15.5:0/1595361432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e641336c0 0x7f8e64133aa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.428+0000 7f8e68b69700 1 -- 172.21.15.5:0/1595361432 >> 172.21.15.5:0/1595361432 conn(0x7f8e640a45d0 msgr2=0x7f8e640a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- 172.21.15.5:0/1595361432 shutdown_connections 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- 172.21.15.5:0/1595361432 wait complete. 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient: monmap: 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 librados: starting msgr at 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 librados: starting objecter 2024-09-16T10:50:06.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 Processor -- start 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- start start 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 librados: setting wanted keys 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 librados: calling monclient init 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient: init 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 0x7f8e641c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-b con 0x7f8e6409db20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 0x7f8e64132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-a con 0x7f8e641336c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e64132e60 0x7f8e6411c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): picked mon.noname-c con 0x7f8e64132e60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e641c8750 con 0x7f8e641336c0 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6409b0d0 con 0x7f8e6409db20 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64133310 con 0x7f8e64132e60 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e68b69700 10 monclient(hunting): authenticate will time out at 865.014476s 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 0x7f8e64132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e64132e60 0x7f8e6411c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 0x7f8e64132920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48996/0 (socket says 172.21.15.5:48996) 2024-09-16T10:50:06.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 1 -- 172.21.15.5:0/2717458210 learned_addr learned my addr 172.21.15.5:0/2717458210 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): get_auth_request con 0x7f8e641336c0 auth_method 0 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 0x7f8e641c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): get_auth_request con 0x7f8e64132e60 auth_method 0 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): get_auth_request con 0x7f8e6409db20 auth_method 0 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 -- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e64132e60 msgr2=0x7f8e6411c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e64132e60 0x7f8e6411c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 -- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 msgr2=0x7f8e64132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 0x7f8e64132920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 monclient: found mon.noname-b 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 -- 172.21.15.5:0/2717458210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580094b0 con 0x7f8e6409db20 2024-09-16T10:50:06.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.432+0000 7f8e637fe700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 0x7f8e641c8210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e58016000 tx=0x7f8e58004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 client.?.objecter ms_handle_connect 0x7f8e6409db20 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801d5b0 con 0x7f8e6409db20 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: dump: 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.457 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5801a4c0 con 0x7f8e6409db20 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801a870 con 0x7f8e6409db20 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 monclient: dump: 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 5 monclient: authenticate success, global_id 4342 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _renew_subs 2024-09-16T10:50:06.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e640a34f0 con 0x7f8e6409db20 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _renew_subs 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e64120000 con 0x7f8e6409db20 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 1 librados: init done 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e617fa700 4 set_mon_vals no callback set 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f8e5801a9d0 con 0x7f8e6409db20 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 --2- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 0x7f8e50039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f8e58050e90 con 0x7f8e6409db20 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 client.4342.objecter ms_dispatch 0x7f8e64000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 3 client.4342.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 3 client.4342.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e44005100 con 0x7f8e6409db20 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 20 client.4342.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e62ffd700 1 --2- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 0x7f8e50039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e62ffd700 10 monclient: get_auth_request con 0x7f8e50036fb0 auth_method 0 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e4bfff700 10 client.4342.objecter ms_handle_connect 0x7f8e50036fb0 2024-09-16T10:50:06.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.440+0000 7f8e62ffd700 1 --2- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 0x7f8e50039470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e6411db80 tx=0x7f8e54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.452+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8e5807cdb0 con 0x7f8e6409db20 2024-09-16T10:50:06.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.452+0000 7f8e4bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.452+0000 7f8e4bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: monmap: 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.473845+0000 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.473845+0000 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 Processor -- start 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 -- start start 2024-09-16T10:50:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: init 2024-09-16T10:50:06.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-b con 0x7f1794135d50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17940a0520 0x7f1794070750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-c con 0x7f17940a0520 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1794070c90 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1794071ac0 con 0x7f1794070c90 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1794136670 con 0x7f17940a0520 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1794135750 con 0x7f1794135d50 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1798e85700 10 monclient(hunting): authenticate will time out at 865.051097s 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1793fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1794070c90 0x7f179413ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59052/0 (socket says 172.21.15.5:59052) 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1793fff700 1 -- 172.21.15.5:0/1137055097 learned_addr learned my addr 172.21.15.5:0/1137055097 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38106/0 (socket says 172.21.15.5:38106) 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): get_auth_request con 0x7f1794135d50 auth_method 0 2024-09-16T10:50:06.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 1 --2- 172.21.15.5:0/1137055097 >> v2:172.21.15.143:3300/0 conn(0x7f17940a0520 0x7f1794070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): get_auth_request con 0x7f17940a0520 auth_method 0 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 -- 172.21.15.5:0/1137055097 >> v2:172.21.15.143:3300/0 conn(0x7f17940a0520 msgr2=0x7f1794070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 --2- 172.21.15.5:0/1137055097 >> v2:172.21.15.143:3300/0 conn(0x7f17940a0520 0x7f1794070750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 -- 172.21.15.5:0/1137055097 >> v1:172.21.15.5:6789/0 conn(0x7f1794070c90 legacy=0x7f179413ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 monclient: found mon.noname-b 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.468+0000 7f17937fe700 1 -- 172.21.15.5:0/1137055097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17940a0100 con 0x7f1794135d50 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17937fe700 1 --2- 172.21.15.5:0/1137055097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f178c009d90 tx=0x7f178c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2e272dc3674fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 1 -- 172.21.15.5:0/1137055097 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c00e070 con 0x7f1794135d50 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: dump: 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.485 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 5 monclient: authenticate success, global_id 4292 2024-09-16T10:50:06.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 1 -- 172.21.15.5:0/1137055097 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c00bb20 con 0x7f1794135d50 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 1 -- 172.21.15.5:0/1137055097 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c012430 con 0x7f1794135d50 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 4 set_mon_vals no callback set 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f17927fc700 10 monclient: dump: 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.487 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: shutdown 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- 172.21.15.5:0/1137055097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 msgr2=0x7f1794136130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- 172.21.15.5:0/1137055097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f178c009d90 tx=0x7f178c00b330 comp rx=0 tx=0).stop 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- 172.21.15.5:0/1137055097 shutdown_connections 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- 172.21.15.5:0/1137055097 >> v2:172.21.15.143:3300/0 conn(0x7f17940a0520 0x7f1794070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- 172.21.15.5:0/1137055097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1794135d50 0x7f1794136130 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- 172.21.15.5:0/1137055097 >> 172.21.15.5:0/1137055097 conn(0x7f17940a45d0 msgr2=0x7f17940a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- 172.21.15.5:0/1137055097 shutdown_connections 2024-09-16T10:50:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- 172.21.15.5:0/1137055097 wait complete. 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: monmap: 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 librados: starting msgr at 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 librados: starting objecter 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 Processor -- start 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- start start 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 librados: setting wanted keys 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 librados: calling monclient init 2024-09-16T10:50:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: init 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1794070c90 0x7f1794133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-a con 0x7f1794070c90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-c con 0x7f17940a0520 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1794135d50 0x7f1794121920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): picked mon.noname-b con 0x7f1794135d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1794124960 con 0x7f1794070c90 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17941cb680 con 0x7f1794135d50 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17941255a0 con 0x7f17940a0520 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.472+0000 7f1798e85700 10 monclient(hunting): authenticate will time out at 865.055992s 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38116/0 (socket says 172.21.15.5:38116) 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 -- 172.21.15.5:0/1311105814 learned_addr learned my addr 172.21.15.5:0/1311105814 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): get_auth_request con 0x7f17940a0520 auth_method 0 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1794070c90 0x7f1794133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): get_auth_request con 0x7f1794070c90 auth_method 0 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1793fff700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1794135d50 0x7f1794121920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 -- 172.21.15.5:0/1311105814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1794135d50 msgr2=0x7f1794121920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1794135d50 0x7f1794121920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 -- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1794070c90 msgr2=0x7f1794133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1794070c90 0x7f1794133e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 -- 172.21.15.5:0/1311105814 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c009a70 con 0x7f17940a0520 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1792ffd700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f178800e9a0 tx=0x7f178800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 client.?.objecter ms_handle_connect 0x7f17940a0520 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788009720 con 0x7f17940a0520 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: dump: 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 5 monclient: authenticate success, global_id 4295 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178800c850 con 0x7f17940a0520 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _renew_subs 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178800cbc0 con 0x7f17940a0520 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1794123a30 con 0x7f17940a0520 2024-09-16T10:50:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 monclient: dump: 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.502 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _renew_subs 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179409a570 con 0x7f17940a0520 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 1 librados: init done 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1791ffb700 4 set_mon_vals no callback set 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f178800cd20 con 0x7f17940a0520 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 --2- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 0x7f177c0395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f1788047ae0 con 0x7f17940a0520 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 10 client.4295.objecter ms_dispatch 0x7f1794000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 3 client.4295.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 3 client.4295.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 10 monclient: monmap: 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.482992+0000 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.482992+0000 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.505 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f177bfff700 20 client.4295.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1780005100 con 0x7f17940a0520 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 1 --2- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 0x7f177c0395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7f17937fe700 10 monclient: get_auth_request con 0x7f177c0370f0 auth_method 0 2024-09-16T10:50:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 1 Processor -- start 2024-09-16T10:50:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 1 -- start start 2024-09-16T10:50:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.476+0000 7ff334dfc700 10 monclient: init 2024-09-16T10:50:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7f177bfff700 10 client.4295.objecter ms_handle_connect 0x7f177c0370f0 2024-09-16T10:50:06.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7f17937fe700 1 --2- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 0x7f177c0395b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f178c014040 tx=0x7f178c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T10:50:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff330135d50 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3300a0520 0x7ff33013aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-a con 0x7ff3300a0520 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff33013afb0 0x7ff33013d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-c con 0x7ff33013afb0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff330071ac0 con 0x7ff330135d50 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff330135750 con 0x7ff3300a0520 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3300a0100 con 0x7ff33013afb0 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff334dfc700 10 monclient(hunting): authenticate will time out at 865.060819s 2024-09-16T10:50:06.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff33013afb0 0x7ff33013d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff33013afb0 0x7ff33013d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:46864/0 (socket says 172.21.15.5:46864) 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 -- 172.21.15.5:0/2152670542 learned_addr learned my addr 172.21.15.5:0/2152670542 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): get_auth_request con 0x7ff33013afb0 auth_method 0 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 -- 172.21.15.5:0/2152670542 >> v2:172.21.15.5:3300/0 conn(0x7ff3300a0520 msgr2=0x7ff33013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 --2- 172.21.15.5:0/2152670542 >> v2:172.21.15.5:3300/0 conn(0x7ff3300a0520 0x7ff33013aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 -- 172.21.15.5:0/2152670542 >> v1:172.21.15.5:6789/0 conn(0x7ff330135d50 legacy=0x7ff330136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 -- 172.21.15.5:0/2152670542 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff330071810 con 0x7ff33013afb0 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.480+0000 7ff32ffff700 1 --2- 172.21.15.5:0/2152670542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff33013afb0 0x7ff33013d3a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff33009fb40 tx=0x7ff320009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcfdd728467d5e61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 1 -- 172.21.15.5:0/2152670542 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320004230 con 0x7ff33013afb0 2024-09-16T10:50:06.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: dump: 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 5 monclient: authenticate success, global_id 4348 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 1 -- 172.21.15.5:0/2152670542 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff320004390 con 0x7ff33013afb0 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff32e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 4 set_mon_vals no callback set 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 10 monclient: shutdown 2024-09-16T10:50:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 -- 172.21.15.5:0/2152670542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff33013afb0 msgr2=0x7ff33013d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 --2- 172.21.15.5:0/2152670542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff33013afb0 0x7ff33013d3a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff33009fb40 tx=0x7ff320009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 -- 172.21.15.5:0/2152670542 shutdown_connections 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 --2- 172.21.15.5:0/2152670542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff33013afb0 0x7ff33013d3a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 --2- 172.21.15.5:0/2152670542 >> v2:172.21.15.5:3300/0 conn(0x7ff3300a0520 0x7ff33013aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.484+0000 7ff334dfc700 1 -- 172.21.15.5:0/2152670542 >> 172.21.15.5:0/2152670542 conn(0x7ff3300a45d0 msgr2=0x7ff3300a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1788017080 con 0x7f17940a0520 2024-09-16T10:50:06.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7f177bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7f177bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- 172.21.15.5:0/2152670542 shutdown_connections 2024-09-16T10:50:06.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- 172.21.15.5:0/2152670542 wait complete. 2024-09-16T10:50:06.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient: monmap: 2024-09-16T10:50:06.520 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 librados: starting msgr at 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 librados: starting objecter 2024-09-16T10:50:06.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 Processor -- start 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- start start 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 librados: setting wanted keys 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 librados: calling monclient init 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient: init 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3300a0520 0x7ff330134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-b con 0x7ff3300a0520 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff330135d50 0x7ff3301345b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-a con 0x7ff330135d50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): picked mon.noname-c con 0x7ff33013afb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33011efc0 con 0x7ff330135d50 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300a30e0 con 0x7ff3300a0520 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33009b0d0 con 0x7ff33013afb0 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff334dfc700 10 monclient(hunting): authenticate will time out at 865.071145s 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38118/0 (socket says 172.21.15.5:38118) 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 -- 172.21.15.5:0/4201457627 learned_addr learned my addr 172.21.15.5:0/4201457627 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): get_auth_request con 0x7ff33013afb0 auth_method 0 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32effd700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff330135d50 0x7ff3301345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32effd700 10 monclient(hunting): get_auth_request con 0x7ff330135d50 auth_method 0 2024-09-16T10:50:06.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3300a0520 0x7ff330134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): get_auth_request con 0x7ff3300a0520 auth_method 0 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-16T10:50:06.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 -- 172.21.15.5:0/4201457627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3300a0520 msgr2=0x7ff330134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3300a0520 0x7ff330134070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 -- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff330135d50 msgr2=0x7ff3301345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff330135d50 0x7ff3301345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.488+0000 7ff32ffff700 1 -- 172.21.15.5:0/4201457627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff320009570 con 0x7ff33013afb0 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.492+0000 7ff32ffff700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3300a0240 tx=0x7ff320004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 10 client.?.objecter ms_handle_connect 0x7ff33013afb0 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32000ba70 con 0x7ff33013afb0 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.496+0000 7ff317fff700 10 monclient: dump: 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.527 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32000bbf0 con 0x7ff33013afb0 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 5 monclient: authenticate success, global_id 4298 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _renew_subs 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3300a3430 con 0x7ff33013afb0 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _renew_subs 2024-09-16T10:50:06.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33011ece0 con 0x7ff33013afb0 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 1 librados: init done 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32001b470 con 0x7ff33013afb0 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 monclient: dump: 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff32dffb700 4 set_mon_vals no callback set 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff320022020 con 0x7ff33013afb0 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 1 --2- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 0x7ff31c039470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7ff320049060 con 0x7ff33013afb0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 client.4298.objecter ms_dispatch 0x7ff330000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 3 client.4298.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 3 client.4298.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 20 client.4298.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff310005100 con 0x7ff33013afb0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff32f7fe700 1 --2- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 0x7ff31c039470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff32f7fe700 10 monclient: get_auth_request con 0x7ff31c036fb0 auth_method 0 2024-09-16T10:50:06.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff317fff700 10 client.4298.objecter ms_handle_connect 0x7ff31c036fb0 2024-09-16T10:50:06.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.500+0000 7ff32f7fe700 1 --2- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 0x7ff31c039470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff328001fa0 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.504+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff320004500 con 0x7ff33013afb0 2024-09-16T10:50:06.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.504+0000 7ff317fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.504+0000 7ff317fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:06.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f501b2cf700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f501b2cf700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f501b2cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5014071ce0 con 0x7f5014138280 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f5012ffd700 1 -- 172.21.15.5:0/3799565645 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5004025940 con 0x7f5014138280 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f5012ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.552+0000 7f5012ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.561 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 10 librados: watch_flush enter 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 10 librados: watch_flush exit 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 20 client.4274.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 10 client.4274.objecter successfully canceled tick 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 msgr2=0x7f5000039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 0x7f5000039420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f500c016040 tx=0x7f500c007000 comp rx=0 tx=0).stop 2024-09-16T10:50:06.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 10 monclient: shutdown 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 msgr2=0x7f5014120f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f501409e5d0 tx=0x7f5004010810 comp rx=0 tx=0).stop 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 shutdown_connections 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5014138280 0x7f5014120f80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5000036f60 0x7f5000039420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5014137ea0 0x7f5014120a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 --2- 172.21.15.5:0/3799565645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ffc003a20 0x7f5014137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 >> 172.21.15.5:0/3799565645 conn(0x7f50140a3200 msgr2=0x7f501413acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 shutdown_connections 2024-09-16T10:50:06.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 -- 172.21.15.5:0/3799565645 wait complete. 2024-09-16T10:50:06.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.556+0000 7f501b2cf700 1 librados: shutdown 2024-09-16T10:50:06.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-16T10:50:06.579 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5bc0121440 con 0x7f5bc00a3510 2024-09-16T10:50:06.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bb67fc700 1 -- 172.21.15.5:0/3687835857 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bb801bb00 con 0x7f5bc00a3510 2024-09-16T10:50:06.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T10:50:06.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bb67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.588 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:06.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 librados: watch_flush enter 2024-09-16T10:50:06.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 librados: watch_flush exit 2024-09-16T10:50:06.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 client.4280.objecter successfully canceled tick 2024-09-16T10:50:06.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 msgr2=0x7f5ba4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 0x7f5ba4039420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0000c00 tx=0x7f5bb0008030 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 10 monclient: shutdown 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 msgr2=0x7f5bc0136550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8007ea0 tx=0x7f5bb800e3c0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 shutdown_connections 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5bc00a3510 0x7f5bc0136550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bc00a2bf0 0x7f5bc0136010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5ba4036f60 0x7f5ba4039420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 --2- 172.21.15.5:0/3687835857 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bc009f780 0x7f5bc01ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.580+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 >> 172.21.15.5:0/3687835857 conn(0x7f5bc00a53e0 msgr2=0x7f5bc00a60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.584+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 shutdown_connections 2024-09-16T10:50:06.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.584+0000 7f5bc6ed2700 1 -- 172.21.15.5:0/3687835857 wait complete. 2024-09-16T10:50:06.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.584+0000 7f5bc6ed2700 1 librados: shutdown 2024-09-16T10:50:06.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-09-16T10:50:06.603 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb981f28700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb981f28700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb981f28700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb95c005680 con 0x7fb97c0a36e0 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb9797fa700 1 -- 172.21.15.5:0/481202021 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb974034090 con 0x7fb97c0a36e0 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb9797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T10:50:06.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.668+0000 7fb9797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.677 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:06.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 10 librados: watch_flush enter 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 10 librados: watch_flush exit 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 20 client.4286.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 10 client.4286.objecter successfully canceled tick 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 msgr2=0x7fb964039560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 0x7fb964039560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb970001940 tx=0x7fb970009040 comp rx=0 tx=0).stop 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 10 monclient: shutdown 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 msgr2=0x7fb97c13aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 0x7fb97c13aa20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb97c074960 tx=0x7fb974004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 shutdown_connections 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb97c0a36e0 0x7fb97c13aa20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb97c0a2bf0 0x7fb97c135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9640370a0 0x7fb964039560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 --2- 172.21.15.5:0/481202021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb97c09f780 0x7fb97c135410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 >> 172.21.15.5:0/481202021 conn(0x7fb97c0a53e0 msgr2=0x7fb97c0a78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 shutdown_connections 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 -- 172.21.15.5:0/481202021 wait complete. 2024-09-16T10:50:06.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.672+0000 7fb981f28700 1 librados: shutdown 2024-09-16T10:50:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff334dfc700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff334dfc700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff334dfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff310005680 con 0x7ff33013afb0 2024-09-16T10:50:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff317fff700 1 -- 172.21.15.5:0/4201457627 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff320034090 con 0x7ff33013afb0 2024-09-16T10:50:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff317fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T10:50:06.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff317fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.685 INFO:teuthology.orchestra.run.smithi005.stdout:47244640261 2024-09-16T10:50:06.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.676+0000 7ff334dfc700 10 librados: watch_flush enter 2024-09-16T10:50:06.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 10 librados: watch_flush exit 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 20 client.4298.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 10 client.4298.objecter successfully canceled tick 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 msgr2=0x7ff31c039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 0x7ff31c039470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff328001fa0 tx=0x7ff328009040 comp rx=0 tx=0).stop 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 10 monclient: shutdown 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 msgr2=0x7ff33011b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff3300a0240 tx=0x7ff320004770 comp rx=0 tx=0).stop 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 shutdown_connections 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff33013afb0 0x7ff33011b010 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff31c036fb0 0x7ff31c039470 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff330135d50 0x7ff3301345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 --2- 172.21.15.5:0/4201457627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3300a0520 0x7ff330134070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 >> 172.21.15.5:0/4201457627 conn(0x7ff3300a45d0 msgr2=0x7ff33009b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 shutdown_connections 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 -- 172.21.15.5:0/4201457627 wait complete. 2024-09-16T10:50:06.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.680+0000 7ff334dfc700 1 librados: shutdown 2024-09-16T10:50:06.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-16T10:50:06.692 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.5 2024-09-16T10:50:06.698 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e68b69700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:06.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e68b69700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:06.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e68b69700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:50:06.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8e44005680 con 0x7f8e6409db20 2024-09-16T10:50:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e4bfff700 1 -- 172.21.15.5:0/2717458210 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e5802c080 con 0x7f8e6409db20 2024-09-16T10:50:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e4bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T10:50:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.704+0000 7f8e4bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.713 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 10 librados: watch_flush enter 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 10 librados: watch_flush exit 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 20 client.4342.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 10 client.4342.objecter successfully canceled tick 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 msgr2=0x7f8e50039470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 0x7f8e50039470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e6411db80 tx=0x7f8e54009040 comp rx=0 tx=0).stop 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 10 monclient: shutdown 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 msgr2=0x7f8e641c8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 0x7f8e641c8210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8e58016000 tx=0x7f8e58004770 comp rx=0 tx=0).stop 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 shutdown_connections 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e64132e60 0x7f8e6411c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e50036fb0 0x7f8e50039470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e641336c0 0x7f8e64132920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 --2- 172.21.15.5:0/2717458210 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e6409db20 0x7f8e641c8210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 >> 172.21.15.5:0/2717458210 conn(0x7f8e640a45d0 msgr2=0x7f8e6406fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 shutdown_connections 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 -- 172.21.15.5:0/2717458210 wait complete. 2024-09-16T10:50:06.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.708+0000 7f8e68b69700 1 librados: shutdown 2024-09-16T10:50:06.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-16T10:50:06.730 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f1798e85700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:06.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f1798e85700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:06.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f1798e85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1780005680 con 0x7f17940a0520 2024-09-16T10:50:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f177bfff700 1 -- 172.21.15.5:0/1311105814 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f178801b3b0 con 0x7f17940a0520 2024-09-16T10:50:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f177bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T10:50:06.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.728+0000 7f177bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:06.737 INFO:teuthology.orchestra.run.smithi005.stdout:47244640259 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 10 librados: watch_flush enter 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 10 librados: watch_flush exit 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 20 client.4295.objecter shutdown clearing up homeless session... 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 10 client.4295.objecter successfully canceled tick 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 msgr2=0x7f177c0395b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 0x7f177c0395b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f178c014040 tx=0x7f178c011000 comp rx=0 tx=0).stop 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 10 monclient: shutdown 2024-09-16T10:50:06.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 msgr2=0x7f17941343a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f178800e9a0 tx=0x7f178800ecb0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 shutdown_connections 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1794135d50 0x7f1794121920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f17940a0520 0x7f17941343a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> v2:172.21.15.143:6812/1098205034 conn(0x7f177c0370f0 0x7f177c0395b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 --2- 172.21.15.5:0/1311105814 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1794070c90 0x7f1794133e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 >> 172.21.15.5:0/1311105814 conn(0x7f17940a45d0 msgr2=0x7f17940a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 shutdown_connections 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 -- 172.21.15.5:0/1311105814 wait complete. 2024-09-16T10:50:06.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.732+0000 7f1798e85700 1 librados: shutdown 2024-09-16T10:50:06.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-16T10:50:06.757 DEBUG:teuthology.parallel:result is None 2024-09-16T10:50:06.757 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T10:50:06.757 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: get_monmap_and_config 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: monmap: 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:06.870592+0000 2024-09-16T10:50:06.873 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:06.870592+0000 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:06.874 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 Processor -- start 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 -- start start 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: init 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T10:50:06.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-b con 0x7f4ed809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ed809d6a0 0x7f4ed809f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed809d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed809dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed8070e40 con 0x7f4ed809dfd0 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80a01d0 con 0x7f4ed809d6a0 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed809f740 con 0x7f4ed809e8e0 2024-09-16T10:50:06.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed759e700 10 monclient(hunting): authenticate will time out at 865.446970s 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38126/0 (socket says 172.21.15.5:38126) 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 -- 172.21.15.5:0/894533580 learned_addr learned my addr 172.21.15.5:0/894533580 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): get_auth_request con 0x7f4ed809e8e0 auth_method 0 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 1 --2- 172.21.15.5:0/894533580 >> v2:172.21.15.5:3300/0 conn(0x7f4ed809d6a0 0x7f4ed809f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): get_auth_request con 0x7f4ed809d6a0 auth_method 0 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ecdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed5d9b700 1 -- 172.21.15.5:0/894533580 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814206050 0 0) 0x7f4ed8070e40 con 0x7f4ed809dfd0 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-16T10:50:06.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 -- 172.21.15.5:0/894533580 >> v2:172.21.15.5:3300/0 conn(0x7f4ed809d6a0 msgr2=0x7f4ed809f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 --2- 172.21.15.5:0/894533580 >> v2:172.21.15.5:3300/0 conn(0x7f4ed809d6a0 0x7f4ed809f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 -- 172.21.15.5:0/894533580 >> v1:172.21.15.5:6789/0 conn(0x7f4ed809dfd0 legacy=0x7f4ed81336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 monclient: found mon.noname-b 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed659c700 1 -- 172.21.15.5:0/894533580 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed809fef0 con 0x7f4ed809e8e0 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.864+0000 7f4ed5d9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- 172.21.15.5:0/894533580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009910 tx=0x7f4ec8009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=336b23d6b04fd78b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 1 -- 172.21.15.5:0/894533580 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8011830 con 0x7f4ed809e8e0 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: dump: 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.889 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 5 monclient: authenticate success, global_id 4301 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 1 -- 172.21.15.5:0/894533580 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec8011990 con 0x7f4ed809e8e0 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:06.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 4 set_mon_vals no callback set 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 1 -- 172.21.15.5:0/894533580 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8011d40 con 0x7f4ed809e8e0 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed5d9b700 10 monclient: dump: 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.891 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: shutdown 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- 172.21.15.5:0/894533580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 msgr2=0x7f4ed809ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- 172.21.15.5:0/894533580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009910 tx=0x7f4ec8009c20 comp rx=0 tx=0).stop 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- 172.21.15.5:0/894533580 shutdown_connections 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- 172.21.15.5:0/894533580 >> v2:172.21.15.5:3300/0 conn(0x7f4ed809d6a0 0x7f4ed809f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- 172.21.15.5:0/894533580 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809e8e0 0x7f4ed809ecc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- 172.21.15.5:0/894533580 >> 172.21.15.5:0/894533580 conn(0x7f4ed80a45d0 msgr2=0x7f4ed80a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- 172.21.15.5:0/894533580 shutdown_connections 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- 172.21.15.5:0/894533580 wait complete. 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: build_initial_monmap 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: monmap: 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:06.893 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 librados: starting msgr at 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 librados: starting objecter 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 Processor -- start 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- start start 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 librados: setting wanted keys 2024-09-16T10:50:06.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 librados: calling monclient init 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: init 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-c con 0x7f4ed809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 0x7f4ed8132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-b con 0x7f4ed809dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ed809e8e0 0x7f4ed81330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed809e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed809b0d0 con 0x7f4ed809e8e0 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8133630 con 0x7f4ed809dfd0 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed809cf70 con 0x7f4ed809d6a0 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed759e700 10 monclient(hunting): authenticate will time out at 865.450895s 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38136/0 (socket says 172.21.15.5:38136) 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ed809e8e0 0x7f4ed81330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 -- 172.21.15.5:0/2426073763 learned_addr learned my addr 172.21.15.5:0/2426073763 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:06.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): get_auth_request con 0x7f4ed809d6a0 auth_method 0 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): get_auth_request con 0x7f4ed809e8e0 auth_method 0 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 0x7f4ed8132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 10 monclient(hunting): get_auth_request con 0x7f4ed809dfd0 auth_method 0 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 -- 172.21.15.5:0/2426073763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 msgr2=0x7f4ed8132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 0x7f4ed8132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 -- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ed809e8e0 msgr2=0x7f4ed81330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ed809e8e0 0x7f4ed81330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 monclient: found mon.noname-c 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 -- 172.21.15.5:0/2426073763 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec8009570 con 0x7f4ed809d6a0 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecdd9b700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 0x7f4ed8132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecf7fe700 10 client.?.objecter ms_handle_connect 0x7f4ed809d6a0 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ecf7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:06.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.868+0000 7f4ed659c700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8011db0 tx=0x7f4ec8004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800b760 con 0x7f4ed809d6a0 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: dump: 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.900 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec801f070 con 0x7f4ed809d6a0 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 5 monclient: authenticate success, global_id 4304 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _renew_subs 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed80a38f0 con 0x7f4ed809d6a0 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800bb00 con 0x7f4ed809d6a0 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 monclient: dump: 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:06.902 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed559a700 4 set_mon_vals no callback set 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _renew_subs 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed8121cf0 con 0x7f4ed809d6a0 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 1 librados: init done 2024-09-16T10:50:06.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed81341b0 con 0x7f4ed809d6a0 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4ec8021030 con 0x7f4ed809d6a0 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 --2- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 0x7f4eb8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f4ec8048a90 con 0x7f4ed809d6a0 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 client.4304.objecter ms_dispatch 0x7f4ed8000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 3 client.4304.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 3 client.4304.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 20 client.4304.objecter dump_active .. 0 homeless 2024-09-16T10:50:06.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecdd9b700 1 --2- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 0x7f4eb8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:06.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecdd9b700 10 monclient: get_auth_request con 0x7f4eb8036f60 auth_method 0 2024-09-16T10:50:06.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecf7fe700 10 client.4304.objecter ms_handle_connect 0x7f4eb8036f60 2024-09-16T10:50:06.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.872+0000 7f4ecdd9b700 1 --2- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 0x7f4eb8039420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8071c20 tx=0x7f4ec0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:06.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.880+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4ec8010380 con 0x7f4ed809d6a0 2024-09-16T10:50:06.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.880+0000 7f4ecf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:06.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:06.880+0000 7f4ecf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:07.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.048+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ed8133dd0 con 0x7f4eb8036f60 2024-09-16T10:50:07.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ecf7fe700 1 -- 172.21.15.5:0/2426073763 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40948 (secure 0 0 0) 0x7f4ec0009990 con 0x7f4eb8036f60 2024-09-16T10:50:07.062 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:07.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 10 librados: watch_flush enter 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 10 librados: watch_flush exit 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 20 client.4304.objecter shutdown clearing up homeless session... 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 10 client.4304.objecter successfully canceled tick 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 msgr2=0x7f4eb8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 0x7f4eb8039420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8071c20 tx=0x7f4ec0008040 comp rx=0 tx=0).stop 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 10 monclient: shutdown 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 msgr2=0x7f4ed81ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8011db0 tx=0x7f4ec8004770 comp rx=0 tx=0).stop 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 shutdown_connections 2024-09-16T10:50:07.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4eb8036f60 0x7f4eb8039420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ed809e8e0 0x7f4ed81330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed809dfd0 0x7f4ed8132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 --2- 172.21.15.5:0/2426073763 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4ed809d6a0 0x7f4ed81ca340 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 >> 172.21.15.5:0/2426073763 conn(0x7f4ed80a45d0 msgr2=0x7f4ed81bbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 shutdown_connections 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 -- 172.21.15.5:0/2426073763 wait complete. 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.056+0000 7f4ed759e700 1 librados: shutdown 2024-09-16T10:50:07.066 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T10:50:07.077 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-16T10:50:05.289423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168092,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566062948,"statfs":{"total":579820584960,"available":579648458752,"internally_reserved":0,"allocated":1900544,"data_stored":1308688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526459+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T10:50:03.526459+0000","last_peered":"2024-09-16T10:50:03.526459+0000","last_clean":"2024-09-16T10:50:03.526459+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T10:50:03.526459+0000","last_undegraded":"2024-09-16T10:50:03.526459+0000","last_fullsized":"2024-09-16T10:50:03.526459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429266+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T10:50:03.429266+0000","last_peered":"2024-09-16T10:50:03.429266+0000","last_clean":"2024-09-16T10:50:03.429266+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T10:50:03.429266+0000","last_undegraded":"2024-09-16T10:50:03.429266+0000","last_fullsized":"2024-09-16T10:50:03.429266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526540+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T10:50:03.526540+0000","last_peered":"2024-09-16T10:50:03.526540+0000","last_clean":"2024-09-16T10:50:03.526540+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T10:50:03.526540+0000","last_undegraded":"2024-09-16T10:50:03.526540+0000","last_fullsized":"2024-09-16T10:50:03.526540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429452+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T10:50:03.429452+0000","last_peered":"2024-09-16T10:50:03.429452+0000","last_clean":"2024-09-16T10:50:03.429452+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T10:50:03.429452+0000","last_undegraded":"2024-09-16T10:50:03.429452+0000","last_fullsized":"2024-09-16T10:50:03.429452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.430870+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T10:50:03.430870+0000","last_peered":"2024-09-16T10:50:03.430870+0000","last_clean":"2024-09-16T10:50:03.430870+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T10:50:03.430870+0000","last_undegraded":"2024-09-16T10:50:03.430870+0000","last_fullsized":"2024-09-16T10:50:03.430870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.688026+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T10:50:03.688026+0000","last_peered":"2024-09-16T10:50:03.688026+0000","last_clean":"2024-09-16T10:50:03.688026+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T10:50:03.688026+0000","last_undegraded":"2024-09-16T10:50:03.688026+0000","last_fullsized":"2024-09-16T10:50:03.688026+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526951+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T10:50:03.526951+0000","last_peered":"2024-09-16T10:50:03.526951+0000","last_clean":"2024-09-16T10:50:03.526951+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T10:50:03.526951+0000","last_undegraded":"2024-09-16T10:50:03.526951+0000","last_fullsized":"2024-09-16T10:50:03.526951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.428485+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T10:50:03.428485+0000","last_peered":"2024-09-16T10:50:03.428485+0000","last_clean":"2024-09-16T10:50:03.428485+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T10:50:03.428485+0000","last_undegraded":"2024-09-16T10:50:03.428485+0000","last_fullsized":"2024-09-16T10:50:03.428485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.525806+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T10:50:03.525806+0000","last_peered":"2024-09-16T10:50:03.525806+0000","last_clean":"2024-09-16T10:50:03.525806+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T10:50:03.525806+0000","last_undegraded":"2024-09-16T10:50:03.525806+0000","last_fullsized":"2024-09-16T10:50:03.525806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":42487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":131072,"data_stored":38359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:50:07.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: get_monmap_and_config 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: monmap: 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:07.189712+0000 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:07.189712+0000 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:07.193 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 Processor -- start 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 -- start start 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: init 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 0x7f92bc09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-b con 0x7f92bc09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92bc09d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f92bc09dfd0 0x7f92bc09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-a con 0x7f92bc09dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc070e40 con 0x7f92bc09d6a0 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0a01d0 con 0x7f92bc09dfd0 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc09f740 con 0x7f92bc09e8e0 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient(hunting): authenticate will time out at 865.766104s 2024-09-16T10:50:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 0x7f92bc09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f92bc09dfd0 0x7f92bc09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f92bc09dfd0 0x7f92bc09f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49042/0 (socket says 172.21.15.5:49042) 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 0x7f92bc09ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38150/0 (socket says 172.21.15.5:38150) 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 -- 172.21.15.5:0/3177604069 learned_addr learned my addr 172.21.15.5:0/3177604069 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): get_auth_request con 0x7f92bc09dfd0 auth_method 0 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): get_auth_request con 0x7f92bc09e8e0 auth_method 0 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 -- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 msgr2=0x7f92bc09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 --2- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 0x7f92bc09ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 -- 172.21.15.5:0/3177604069 >> v1:172.21.15.143:6789/0 conn(0x7f92bc09d6a0 legacy=0x7f92bc1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 monclient: found mon.noname-a 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 -- 172.21.15.5:0/3177604069 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc09fef0 con 0x7f92bc09dfd0 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2456700 1 --2- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09dfd0 0x7f92bc09f200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e50 tx=0x7f92b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958a3a51f5cf36a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 1 -- 172.21.15.5:0/3177604069 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400e070 con 0x7f92bc09dfd0 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: dump: 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 5 monclient: authenticate success, global_id 4377 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 1 -- 172.21.15.5:0/3177604069 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400b910 con 0x7f92bc09dfd0 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c2c57700 4 set_mon_vals no callback set 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 1 -- 172.21.15.5:0/3177604069 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400bcc0 con 0x7f92bc09dfd0 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.184+0000 7f92c0c53700 10 monclient: dump: 2024-09-16T10:50:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: shutdown 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09dfd0 msgr2=0x7f92bc09f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09dfd0 0x7f92bc09f200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e50 tx=0x7f92b400b330 comp rx=0 tx=0).stop 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- 172.21.15.5:0/3177604069 shutdown_connections 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09dfd0 0x7f92bc09f200 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- 172.21.15.5:0/3177604069 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09e8e0 0x7f92bc09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- 172.21.15.5:0/3177604069 >> 172.21.15.5:0/3177604069 conn(0x7f92bc0a45d0 msgr2=0x7f92bc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- 172.21.15.5:0/3177604069 shutdown_connections 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- 172.21.15.5:0/3177604069 wait complete. 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: monmap: 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.206 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 librados: starting msgr at 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 librados: starting objecter 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 Processor -- start 2024-09-16T10:50:07.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- start start 2024-09-16T10:50:07.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 librados: setting wanted keys 2024-09-16T10:50:07.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 librados: calling monclient init 2024-09-16T10:50:07.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: init 2024-09-16T10:50:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-a con 0x7f92bc09d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09dfd0 0x7f92bc1cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-c con 0x7f92bc09dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc09e8e0 0x7f92bc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): picked mon.noname-b con 0x7f92bc09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc11f8e0 con 0x7f92bc09d6a0 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0a3b30 con 0x7f92bc09e8e0 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0a3800 con 0x7f92bc09dfd0 2024-09-16T10:50:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c2c57700 10 monclient(hunting): authenticate will time out at 865.771794s 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c1c55700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.188+0000 7f92c1454700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09dfd0 0x7f92bc1cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49056/0 (socket says 172.21.15.5:49056) 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 -- 172.21.15.5:0/643663120 learned_addr learned my addr 172.21.15.5:0/643663120 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): get_auth_request con 0x7f92bc09d6a0 auth_method 0 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): get_auth_request con 0x7f92bc09dfd0 auth_method 0 2024-09-16T10:50:07.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc09e8e0 0x7f92bc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): get_auth_request con 0x7f92bc09e8e0 auth_method 0 2024-09-16T10:50:07.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2456700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 -- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09dfd0 msgr2=0x7f92bc1cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09dfd0 0x7f92bc1cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 -- 172.21.15.5:0/643663120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc09e8e0 msgr2=0x7f92bc06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc09e8e0 0x7f92bc06ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 monclient: found mon.noname-a 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 -- 172.21.15.5:0/643663120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009b30 con 0x7f92bc09d6a0 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1c55700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e9d0 tx=0x7f92ac00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 client.?.objecter ms_handle_connect 0x7f92bc09d6a0 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac009720 con 0x7f92bc09d6a0 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: dump: 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 5 monclient: authenticate success, global_id 4380 2024-09-16T10:50:07.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac00ee70 con 0x7f92bc09d6a0 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _renew_subs 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc11ed10 con 0x7f92bc09d6a0 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00ca50 con 0x7f92bc09d6a0 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b3fff700 4 set_mon_vals no callback set 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 monclient: dump: 2024-09-16T10:50:07.223 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _renew_subs 2024-09-16T10:50:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc120d90 con 0x7f92bc09d6a0 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 1 librados: init done 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f92ac00cbb0 con 0x7f92bc09d6a0 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 --2- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 0x7f92a0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f92ac047720 con 0x7f92bc09d6a0 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 10 client.4380.objecter ms_dispatch 0x7f92bc000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 3 client.4380.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 3 client.4380.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc0a3650 con 0x7f92bc09d6a0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92b27fc700 20 client.4380.objecter dump_active .. 0 homeless 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 1 --2- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 0x7f92a0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.192+0000 7f92c1454700 10 monclient: get_auth_request con 0x7f92a0036f60 auth_method 0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.196+0000 7f92b27fc700 10 client.4380.objecter ms_handle_connect 0x7f92a0036f60 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.196+0000 7f92c1454700 1 --2- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 0x7f92a0039420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f92bc070680 tx=0x7f92b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.200+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f92ac017080 con 0x7f92bc09d6a0 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.200+0000 7f92b27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.200+0000 7f92b27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.376+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92bc09bee0 con 0x7f92a0036f60 2024-09-16T10:50:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92b27fc700 1 -- 172.21.15.5:0/643663120 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40948 (secure 0 0 0) 0x7f92bc09bee0 con 0x7f92a0036f60 2024-09-16T10:50:07.388 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 10 librados: watch_flush enter 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 10 librados: watch_flush exit 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 20 client.4380.objecter shutdown clearing up homeless session... 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 10 client.4380.objecter successfully canceled tick 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 msgr2=0x7f92a0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 0x7f92a0039420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f92bc070680 tx=0x7f92b8009040 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 10 monclient: shutdown 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 msgr2=0x7f92bc1cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e9d0 tx=0x7f92ac00ece0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 shutdown_connections 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc09e8e0 0x7f92bc06ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f92bc09dfd0 0x7f92bc1cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92a0036f60 0x7f92a0039420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 --2- 172.21.15.5:0/643663120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f92bc09d6a0 0x7f92bc1cb1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 >> 172.21.15.5:0/643663120 conn(0x7f92bc0a45d0 msgr2=0x7f92bc09c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 shutdown_connections 2024-09-16T10:50:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 -- 172.21.15.5:0/643663120 wait complete. 2024-09-16T10:50:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.380+0000 7f92c2c57700 1 librados: shutdown 2024-09-16T10:50:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T10:50:07.403 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-16T10:50:07.290025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168092,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566062948,"statfs":{"total":579820584960,"available":579648458752,"internally_reserved":0,"allocated":1900544,"data_stored":1308688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526459+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T10:50:03.526459+0000","last_peered":"2024-09-16T10:50:03.526459+0000","last_clean":"2024-09-16T10:50:03.526459+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T10:50:03.526459+0000","last_undegraded":"2024-09-16T10:50:03.526459+0000","last_fullsized":"2024-09-16T10:50:03.526459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429266+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T10:50:03.429266+0000","last_peered":"2024-09-16T10:50:03.429266+0000","last_clean":"2024-09-16T10:50:03.429266+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T10:50:03.429266+0000","last_undegraded":"2024-09-16T10:50:03.429266+0000","last_fullsized":"2024-09-16T10:50:03.429266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526540+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T10:50:03.526540+0000","last_peered":"2024-09-16T10:50:03.526540+0000","last_clean":"2024-09-16T10:50:03.526540+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T10:50:03.526540+0000","last_undegraded":"2024-09-16T10:50:03.526540+0000","last_fullsized":"2024-09-16T10:50:03.526540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429452+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T10:50:03.429452+0000","last_peered":"2024-09-16T10:50:03.429452+0000","last_clean":"2024-09-16T10:50:03.429452+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T10:50:03.429452+0000","last_undegraded":"2024-09-16T10:50:03.429452+0000","last_fullsized":"2024-09-16T10:50:03.429452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.430870+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T10:50:03.430870+0000","last_peered":"2024-09-16T10:50:03.430870+0000","last_clean":"2024-09-16T10:50:03.430870+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T10:50:03.430870+0000","last_undegraded":"2024-09-16T10:50:03.430870+0000","last_fullsized":"2024-09-16T10:50:03.430870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.688026+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T10:50:03.688026+0000","last_peered":"2024-09-16T10:50:03.688026+0000","last_clean":"2024-09-16T10:50:03.688026+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T10:50:03.688026+0000","last_undegraded":"2024-09-16T10:50:03.688026+0000","last_fullsized":"2024-09-16T10:50:03.688026+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526951+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T10:50:03.526951+0000","last_peered":"2024-09-16T10:50:03.526951+0000","last_clean":"2024-09-16T10:50:03.526951+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T10:50:03.526951+0000","last_undegraded":"2024-09-16T10:50:03.526951+0000","last_fullsized":"2024-09-16T10:50:03.526951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.428485+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T10:50:03.428485+0000","last_peered":"2024-09-16T10:50:03.428485+0000","last_clean":"2024-09-16T10:50:03.428485+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T10:50:03.428485+0000","last_undegraded":"2024-09-16T10:50:03.428485+0000","last_fullsized":"2024-09-16T10:50:03.428485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.525806+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T10:50:03.525806+0000","last_peered":"2024-09-16T10:50:03.525806+0000","last_clean":"2024-09-16T10:50:03.525806+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T10:50:03.525806+0000","last_undegraded":"2024-09-16T10:50:03.525806+0000","last_fullsized":"2024-09-16T10:50:03.525806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":42487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":131072,"data_stored":38359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:50:07.405 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T10:50:07.405 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T10:50:07.405 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T10:50:07.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T10:50:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 10 monclient: get_monmap_and_config 2024-09-16T10:50:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 10 monclient: monmap: 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:07.511928+0000 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:07.511928+0000 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 1 Processor -- start 2024-09-16T10:50:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 1 -- start start 2024-09-16T10:50:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.504+0000 7fbe11f08700 10 monclient: init 2024-09-16T10:50:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T10:50:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbe0c09e8e0 0x7fbe0c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c09e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe0c09dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c070e40 con 0x7fbe0c09dfd0 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0a01d0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c09f740 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient(hunting): authenticate will time out at 866.088233s 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11707700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fbe0c09dfd0 0x7fbe0c1336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59086/0 (socket says 172.21.15.5:59086) 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11707700 1 -- 172.21.15.5:0/3594949712 learned_addr learned my addr 172.21.15.5:0/3594949712 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 --2- 172.21.15.5:0/3594949712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): get_auth_request con 0x7fbe0c09d6a0 auth_method 0 2024-09-16T10:50:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 1 --2- 172.21.15.5:0/3594949712 >> v2:172.21.15.143:3300/0 conn(0x7fbe0c09e8e0 0x7fbe0c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): get_auth_request con 0x7fbe0c09e8e0 auth_method 0 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574242685 0 0) 0x7fbe0c070e40 con 0x7fbe0c09dfd0 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient(hunting): _negotiate 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient(hunting): my global_id is 4383 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf80035d0 con 0x7fbe0c09dfd0 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 -- 172.21.15.5:0/3594949712 >> v2:172.21.15.143:3300/0 conn(0x7fbe0c09e8e0 msgr2=0x7fbe0c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 --2- 172.21.15.5:0/3594949712 >> v2:172.21.15.143:3300/0 conn(0x7fbe0c09e8e0 0x7fbe0c09ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199945508 0 0) 0x7fbdf80035d0 con 0x7fbe0c09dfd0 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 -- 172.21.15.5:0/3594949712 >> v1:172.21.15.5:6789/0 conn(0x7fbe0c09dfd0 legacy=0x7fbe0c1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 monclient: found mon.noname-b 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 -- 172.21.15.5:0/3594949712 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c09fef0 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0bfff700 1 --2- 172.21.15.5:0/3594949712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c09f200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009d90 tx=0x7fbdf400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80025bfe35c3e045 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf400e070 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: dump: 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4003a40 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 1 -- 172.21.15.5:0/3594949712 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04001670 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 5 monclient: authenticate success, global_id 4313 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 4 set_mon_vals no callback set 2024-09-16T10:50:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe0b7fe700 10 monclient: dump: 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 10 monclient: shutdown 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- 172.21.15.5:0/3594949712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 msgr2=0x7fbe0c09f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 --2- 172.21.15.5:0/3594949712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c09f200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009d90 tx=0x7fbdf400b330 comp rx=0 tx=0).stop 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- 172.21.15.5:0/3594949712 shutdown_connections 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 --2- 172.21.15.5:0/3594949712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c09f200 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 --2- 172.21.15.5:0/3594949712 >> v2:172.21.15.143:3300/0 conn(0x7fbe0c09e8e0 0x7fbe0c09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- 172.21.15.5:0/3594949712 >> 172.21.15.5:0/3594949712 conn(0x7fbe0c0a45d0 msgr2=0x7fbe0c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- 172.21.15.5:0/3594949712 shutdown_connections 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.508+0000 7fbe11f08700 1 -- 172.21.15.5:0/3594949712 wait complete. 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: monmap: 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 librados: starting msgr at 2024-09-16T10:50:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 librados: starting objecter 2024-09-16T10:50:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 Processor -- start 2024-09-16T10:50:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- start start 2024-09-16T10:50:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 librados: setting wanted keys 2024-09-16T10:50:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 librados: calling monclient init 2024-09-16T10:50:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: init 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe0c09dfd0 0x7fbe0c132f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c09dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c09e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c11fee0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c09b0d0 con 0x7fbe0c09dfd0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0a39e0 con 0x7fbe0c09d6a0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient(hunting): authenticate will time out at 866.094150s 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49064/0 (socket says 172.21.15.5:49064) 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 -- 172.21.15.5:0/2836861890 learned_addr learned my addr 172.21.15.5:0/2836861890 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): get_auth_request con 0x7fbe0c09e8e0 auth_method 0 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c1329e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38174/0 (socket says 172.21.15.5:38174) 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): get_auth_request con 0x7fbe0c09d6a0 auth_method 0 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe0c09dfd0 0x7fbe0c132f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): get_auth_request con 0x7fbe0c09dfd0 auth_method 0 2024-09-16T10:50:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe10f06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 -- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 msgr2=0x7fbe0c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c1329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 -- 172.21.15.5:0/2836861890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe0c09dfd0 msgr2=0x7fbe0c132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe0c09dfd0 0x7fbe0c132f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 monclient: found mon.noname-a 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 -- 172.21.15.5:0/2836861890 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4009a70 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11707700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400c6d0 tx=0x7fbe0400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 client.?.objecter ms_handle_connect 0x7fbe0c09e8e0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04004ca0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: dump: 2024-09-16T10:50:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe04004e00 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 5 monclient: authenticate success, global_id 4386 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe040072d0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _renew_subs 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c133750 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe0affd700 4 set_mon_vals no callback set 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe097fa700 10 monclient: dump: 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _renew_subs 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c1203a0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.512+0000 7fbe11f08700 1 librados: init done 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fbe04007430 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 1 --2- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 0x7fbdec039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe11f08700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7fbe040474b0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 10 client.4386.objecter ms_dispatch 0x7fbe0c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 3 client.4386.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 3 client.4386.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe11f08700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe11f08700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c11e980 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 20 client.4386.objecter dump_active .. 0 homeless 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe10f06700 1 --2- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 0x7fbdec039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe10f06700 10 monclient: get_auth_request con 0x7fbdec036f60 auth_method 0 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe097fa700 10 client.4386.objecter ms_handle_connect 0x7fbdec036f60 2024-09-16T10:50:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.516+0000 7fbe10f06700 1 --2- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 0x7fbdec039420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009bf0 tx=0x7fbdfc009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.528+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbe040076e0 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.528+0000 7fbe097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.528+0000 7fbe097fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe11f08700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T10:50:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe11f08700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T10:50:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe11f08700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe0c09fa20 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe097fa700 1 -- 172.21.15.5:0/2836861890 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbe0401bb30 con 0x7fbe0c09e8e0 2024-09-16T10:50:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T10:50:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.728+0000 7fbe097fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:07.738 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:07.738 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T10:50:07.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 10 librados: watch_flush enter 2024-09-16T10:50:07.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 10 librados: watch_flush exit 2024-09-16T10:50:07.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 20 client.4386.objecter shutdown clearing up homeless session... 2024-09-16T10:50:07.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 10 client.4386.objecter successfully canceled tick 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 msgr2=0x7fbdec039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 0x7fbdec039420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009bf0 tx=0x7fbdfc009240 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 10 monclient: shutdown 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 msgr2=0x7fbe0c11c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400c6d0 tx=0x7fbe0400c9e0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 shutdown_connections 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbdec036f60 0x7fbdec039420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbe0c09e8e0 0x7fbe0c11c510 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe0c09dfd0 0x7fbe0c132f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 --2- 172.21.15.5:0/2836861890 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbe0c09d6a0 0x7fbe0c1329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 >> 172.21.15.5:0/2836861890 conn(0x7fbe0c0a45d0 msgr2=0x7fbe0c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 shutdown_connections 2024-09-16T10:50:07.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 -- 172.21.15.5:0/2836861890 wait complete. 2024-09-16T10:50:07.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.732+0000 7fbe11f08700 1 librados: shutdown 2024-09-16T10:50:07.752 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T10:50:07.753 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T10:50:07.766 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T10:50:07.766 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:50:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.876+0000 7f8ea1f78700 10 monclient: get_monmap_and_config 2024-09-16T10:50:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.876+0000 7f8ea1f78700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.876+0000 7f8ea1f78700 10 monclient: monmap: 2024-09-16T10:50:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.882 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:07.879773+0000 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:07.879773+0000 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:07.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 Processor -- start 2024-09-16T10:50:07.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- start start 2024-09-16T10:50:07.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: init 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e9c0a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e9c09f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 0x7f8e9c13e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a36e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a3c40 con 0x7f8e9c09f780 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a4300 con 0x7f8e9c0a2bf0 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c09fda0 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): authenticate will time out at 866.456941s 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8e9c09f780 0x7f8e9c13ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59102/0 (socket says 172.21.15.5:59102) 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 1 -- 172.21.15.5:0/1327542883 learned_addr learned my addr 172.21.15.5:0/1327542883 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 --2- 172.21.15.5:0/1327542883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 0x7f8e9c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): get_auth_request con 0x7f8e9c0a36e0 auth_method 0 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074807157 0 0) 0x7f8e9c0a3c40 con 0x7f8e9c09f780 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): my global_id is 4389 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c0035d0 con 0x7f8e9c09f780 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6000491 0 0) 0x7f8e9c0a4300 con 0x7f8e9c0a2bf0 2024-09-16T10:50:07.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _negotiate 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient(hunting): my global_id is 4369 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c0a3c40 con 0x7f8e9c0a2bf0 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 961949761 0 0) 0x7f8e8c0035d0 con 0x7f8e9c09f780 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 -- 172.21.15.5:0/1327542883 >> v1:172.21.15.143:6789/0 conn(0x7f8e9c0a2bf0 legacy=0x7f8e9c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 -- 172.21.15.5:0/1327542883 >> v1:172.21.15.5:6789/0 conn(0x7f8e9c09f780 legacy=0x7f8e9c13ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 monclient: found mon.noname-b 2024-09-16T10:50:07.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 -- 172.21.15.5:0/1327542883 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0a0080 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 --2- 172.21.15.5:0/1327542883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 0x7f8e9c13e530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009dc0 tx=0x7f8e9400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2629badc62e7297 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400e070 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: dump: 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 5 monclient: authenticate success, global_id 4319 2024-09-16T10:50:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94003a40 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 4 set_mon_vals no callback set 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 1 -- 172.21.15.5:0/1327542883 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e88001670 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9b7fe700 10 monclient: dump: 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: shutdown 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1327542883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 msgr2=0x7f8e9c13e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1327542883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 0x7f8e9c13e530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009dc0 tx=0x7f8e9400b330 comp rx=0 tx=0).stop 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1327542883 shutdown_connections 2024-09-16T10:50:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1327542883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a36e0 0x7f8e9c13e530 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1327542883 >> 172.21.15.5:0/1327542883 conn(0x7f8e9c0a53e0 msgr2=0x7f8e9c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1327542883 shutdown_connections 2024-09-16T10:50:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1327542883 wait complete. 2024-09-16T10:50:07.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: build_initial_monmap 2024-09-16T10:50:07.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: monmap: 2024-09-16T10:50:07.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:07.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 librados: starting msgr at 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 librados: starting objecter 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 Processor -- start 2024-09-16T10:50:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- start start 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 librados: setting wanted keys 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 librados: calling monclient init 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: init 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c09f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c0a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e9c0a36e0 0x7f8e9c13a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a36e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c13ce40 con 0x7f8e9c09f780 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c11ebc0 con 0x7f8e9c0a36e0 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c11eea0 con 0x7f8e9c0a2bf0 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1f78700 10 monclient(hunting): authenticate will time out at 866.462943s 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49068/0 (socket says 172.21.15.5:49068) 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38190/0 (socket says 172.21.15.5:38190) 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 -- 172.21.15.5:0/1132135019 learned_addr learned my addr 172.21.15.5:0/1132135019 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:07.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): get_auth_request con 0x7f8e9c09f780 auth_method 0 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): get_auth_request con 0x7f8e9c0a2bf0 auth_method 0 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e9c0a36e0 0x7f8e9c13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): get_auth_request con 0x7f8e9c0a36e0 auth_method 0 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:07.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea1777700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 -- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 msgr2=0x7f8e9c1cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 -- 172.21.15.5:0/1132135019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e9c0a36e0 msgr2=0x7f8e9c13a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e9c0a36e0 0x7f8e9c13a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient: _finish_hunting 0 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 monclient: found mon.noname-a 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8ea0f76700 1 -- 172.21.15.5:0/1132135019 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e94009a70 con 0x7f8e9c09f780 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.880+0000 7f8e9bfff700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea0f76700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000c730 tx=0x7f8e9000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 client.?.objecter ms_handle_connect 0x7f8e9c09f780 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e900052d0 con 0x7f8e9c09f780 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: dump: 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 5 monclient: authenticate success, global_id 4392 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _renew_subs 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90005450 con 0x7f8e9c09f780 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c13c420 con 0x7f8e9c09f780 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e900038e0 con 0x7f8e9c09f780 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 monclient: dump: 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e9affd700 4 set_mon_vals no callback set 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _renew_subs 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c0a7740 con 0x7f8e9c09f780 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 1 librados: init done 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f8e90003a40 con 0x7f8e9c09f780 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c11e8e0 con 0x7f8e9c09f780 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 --2- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 0x7f8e80039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f8e90046b90 con 0x7f8e9c09f780 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 client.4392.objecter ms_dispatch 0x7f8e9c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 3 client.4392.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 3 client.4392.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 20 client.4392.objecter dump_active .. 0 homeless 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e9bfff700 1 --2- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 0x7f8e80039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e9bfff700 10 monclient: get_auth_request con 0x7f8e80036f60 auth_method 0 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e997fa700 10 client.4392.objecter ms_handle_connect 0x7f8e80036f60 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.884+0000 7f8e9bfff700 1 --2- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 0x7f8e80039420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e88000f60 tx=0x7f8e88001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.896+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8e90019d00 con 0x7f8e9c09f780 2024-09-16T10:50:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.896+0000 7f8e997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:07.896+0000 7f8e997fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8ea1f78700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8ea1f78700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8ea1f78700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e9c0a2fd0 con 0x7f8e9c09f780 2024-09-16T10:50:08.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8e997fa700 1 -- 172.21.15.5:0/1132135019 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10822 (secure 0 0 0) 0x7f8e90017090 con 0x7f8e9c09f780 2024-09-16T10:50:08.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8e997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T10:50:08.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.068+0000 7f8e997fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:08.079 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:08.079 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":16,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T10:50:03.409895+0000","last_up_change":"2024-09-16T10:49:58.356658+0000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:49:59.343692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:49:59.729840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2408116882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":2408116882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":2408116882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":2408116882}]},"public_addr":"172.21.15.5:6808/2408116882","cluster_addr":"172.21.15.5:6809/2408116882","heartbeat_back_addr":"172.21.15.5:6811/2408116882","heartbeat_front_addr":"172.21.15.5:6810/2408116882","state":["exists","up"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3675136489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":3675136489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":3675136489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3675136489}]},"public_addr":"172.21.15.5:6800/3675136489","cluster_addr":"172.21.15.5:6801/3675136489","heartbeat_back_addr":"172.21.15.5:6803/3675136489","heartbeat_front_addr":"172.21.15.5:6802/3675136489","state":["exists","up"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2015722671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":2015722671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":2015722671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2015722671}]},"public_addr":"172.21.15.5:6804/2015722671","cluster_addr":"172.21.15.5:6805/2015722671","heartbeat_back_addr":"172.21.15.5:6807/2015722671","heartbeat_front_addr":"172.21.15.5:6806/2015722671","state":["exists","up"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3342511476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6801","nonce":3342511476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6803","nonce":3342511476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3342511476}]},"public_addr":"172.21.15.143:6800/3342511476","cluster_addr":"172.21.15.143:6801/3342511476","heartbeat_back_addr":"172.21.15.143:6803/3342511476","heartbeat_front_addr":"172.21.15.143:6802/3342511476","state":["exists","up"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3348750750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":3348750750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6811","nonce":3348750750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3348750750}]},"public_addr":"172.21.15.143:6808/3348750750","cluster_addr":"172.21.15.143:6809/3348750750","heartbeat_back_addr":"172.21.15.143:6811/3348750750","heartbeat_front_addr":"172.21.15.143:6810/3348750750","state":["exists","up"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3538568827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6805","nonce":3538568827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6807","nonce":3538568827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3538568827}]},"public_addr":"172.21.15.143:6804/3538568827","cluster_addr":"172.21.15.143:6805/3538568827","heartbeat_back_addr":"172.21.15.143:6807/3538568827","heartbeat_front_addr":"172.21.15.143:6806/3538568827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.912822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.644997+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.809738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.543973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.760941+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.643053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 10 librados: watch_flush enter 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 10 librados: watch_flush exit 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 20 client.4392.objecter shutdown clearing up homeless session... 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 10 client.4392.objecter successfully canceled tick 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 msgr2=0x7f8e80039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 0x7f8e80039420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8e88000f60 tx=0x7f8e88001e00 comp rx=0 tx=0).stop 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 10 monclient: shutdown 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 msgr2=0x7f8e9c1cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000c730 tx=0x7f8e9000ca40 comp rx=0 tx=0).stop 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 shutdown_connections 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e9c0a36e0 0x7f8e9c13a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8e9c0a2bf0 0x7f8e9c1cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8e80036f60 0x7f8e80039420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 --2- 172.21.15.5:0/1132135019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e9c09f780 0x7f8e9c1cabe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 >> 172.21.15.5:0/1132135019 conn(0x7f8e9c0a53e0 msgr2=0x7f8e9c06eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 shutdown_connections 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 -- 172.21.15.5:0/1132135019 wait complete. 2024-09-16T10:50:08.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.072+0000 7f8ea1f78700 1 librados: shutdown 2024-09-16T10:50:08.094 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T10:49:59.343692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T10:49:59.729840+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T10:50:08.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: get_monmap_and_config 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: monmap: 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:08.209566+0000 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:08.209566+0000 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.212 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 Processor -- start 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 -- start start 2024-09-16T10:50:08.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: init 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9700a38d0 0x7fd970135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9700a38d0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9700a21c0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 0x7fd970132910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fd9700a25d0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:08.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700a3cb0 con 0x7fd9700a21c0 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700a29e0 con 0x7fd9700a38d0 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700a35a0 con 0x7fd9700a25d0 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96f59e700 10 monclient(hunting): authenticate will time out at 866.787532s 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9700a38d0 0x7fd970135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9700a38d0 0x7fd970135bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49082/0 (socket says 172.21.15.5:49082) 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 1 -- 172.21.15.5:0/11786747 learned_addr learned my addr 172.21.15.5:0/11786747 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): get_auth_request con 0x7fd9700a38d0 auth_method 0 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96ed9d700 1 --2- 172.21.15.5:0/11786747 >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 0x7fd970132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96ed9d700 10 monclient(hunting): get_auth_request con 0x7fd9700a25d0 auth_method 0 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.204+0000 7fd96ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 10 monclient(hunting): handle_auth_done global_id 4395 payload 274 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 -- 172.21.15.5:0/11786747 >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 msgr2=0x7fd970132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 --2- 172.21.15.5:0/11786747 >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 0x7fd970132910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 -- 172.21.15.5:0/11786747 >> v1:172.21.15.143:6789/0 conn(0x7fd9700a21c0 legacy=0x7fd970132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 monclient: found mon.noname-a 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 -- 172.21.15.5:0/11786747 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97009f9e0 con 0x7fd9700a38d0 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96ed9d700 1 --2- 172.21.15.5:0/11786747 >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 0x7fd970132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96e59c700 1 --2- 172.21.15.5:0/11786747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a38d0 0x7fd970135bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd97006b180 tx=0x7fd960009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df9b43a11687939 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 1 -- 172.21.15.5:0/11786747 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96000b8a0 con 0x7fd9700a38d0 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: dump: 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.219 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 5 monclient: authenticate success, global_id 4395 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 1 -- 172.21.15.5:0/11786747 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96000ba00 con 0x7fd9700a38d0 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 4 set_mon_vals no callback set 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 1 -- 172.21.15.5:0/11786747 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96000bd90 con 0x7fd9700a38d0 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96dd9b700 10 monclient: dump: 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 10 monclient: shutdown 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 -- 172.21.15.5:0/11786747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a38d0 msgr2=0x7fd970135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 --2- 172.21.15.5:0/11786747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a38d0 0x7fd970135bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd97006b180 tx=0x7fd960009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 -- 172.21.15.5:0/11786747 shutdown_connections 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 --2- 172.21.15.5:0/11786747 >> v2:172.21.15.143:3300/0 conn(0x7fd9700a25d0 0x7fd970132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 --2- 172.21.15.5:0/11786747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a38d0 0x7fd970135bc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 -- 172.21.15.5:0/11786747 >> 172.21.15.5:0/11786747 conn(0x7fd9700a8440 msgr2=0x7fd9700a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 -- 172.21.15.5:0/11786747 shutdown_connections 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 -- 172.21.15.5:0/11786747 wait complete. 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 10 monclient: monmap: 2024-09-16T10:50:08.227 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 librados: starting msgr at 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.208+0000 7fd96f59e700 1 librados: starting objecter 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 Processor -- start 2024-09-16T10:50:08.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- start start 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 librados: setting wanted keys 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 librados: calling monclient init 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: init 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9700a21c0 0x7fd9701cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fd9700a21c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 0x7fd97006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9700a25d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fd9700a38d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970120410 con 0x7fd9700a25d0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd970121060 con 0x7fd9700a21c0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700a7ba0 con 0x7fd9700a38d0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient(hunting): authenticate will time out at 866.792673s 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 0x7fd97006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 0x7fd97006f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49088/0 (socket says 172.21.15.5:49088) 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38198/0 (socket says 172.21.15.5:38198) 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 1 -- 172.21.15.5:0/399629176 learned_addr learned my addr 172.21.15.5:0/399629176 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): get_auth_request con 0x7fd9700a25d0 auth_method 0 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): get_auth_request con 0x7fd9700a38d0 auth_method 0 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96e59c700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9700a21c0 0x7fd9701cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96e59c700 10 monclient(hunting): get_auth_request con 0x7fd9700a21c0 auth_method 0 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd965d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 -- 172.21.15.5:0/399629176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9700a21c0 msgr2=0x7fd9701cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9700a21c0 0x7fd9701cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 -- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 msgr2=0x7fd97006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 0x7fd97006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 monclient: found mon.noname-c 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 -- 172.21.15.5:0/399629176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960009570 con 0x7fd9700a38d0 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96ed9d700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd968007f20 tx=0x7fd96800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 client.?.objecter ms_handle_connect 0x7fd9700a38d0 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96800de40 con 0x7fd9700a38d0 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: dump: 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.233 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96800f040 con 0x7fd9700a38d0 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 5 monclient: authenticate success, global_id 4325 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _renew_subs 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97012fb20 con 0x7fd9700a38d0 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968003c80 con 0x7fd9700a38d0 2024-09-16T10:50:08.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 monclient: dump: 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96cd99700 4 set_mon_vals no callback set 2024-09-16T10:50:08.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _renew_subs 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9700710c0 con 0x7fd9700a38d0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 librados: init done 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd968004030 con 0x7fd9700a38d0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 --2- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 0x7fd950039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7fd968047800 con 0x7fd9700a38d0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 10 client.4325.objecter ms_dispatch 0x7fd970000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 3 client.4325.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 3 client.4325.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970075890 con 0x7fd9700a38d0 2024-09-16T10:50:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.212+0000 7fd9677fe700 20 client.4325.objecter dump_active .. 0 homeless 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.216+0000 7fd96e59c700 1 --2- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 0x7fd950039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.216+0000 7fd96e59c700 10 monclient: get_auth_request con 0x7fd950036f60 auth_method 0 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.216+0000 7fd9677fe700 10 client.4325.objecter ms_handle_connect 0x7fd950036f60 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.216+0000 7fd96e59c700 1 --2- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 0x7fd950039420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd960008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.220+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd96801a020 con 0x7fd9700a38d0 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.220+0000 7fd9677fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.220+0000 7fd9677fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:08.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.392+0000 7fd96f59e700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:50:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.392+0000 7fd96f59e700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:50:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.392+0000 7fd96f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.392+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd970135bc0 con 0x7fd9700a38d0 2024-09-16T10:50:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.396+0000 7fd9677fe700 1 -- 172.21.15.5:0/399629176 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd96801d360 con 0x7fd9700a38d0 2024-09-16T10:50:08.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.396+0000 7fd9677fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T10:50:08.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.396+0000 7fd9677fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:08.404 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-09-16T10:50:08.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 10 librados: watch_flush enter 2024-09-16T10:50:08.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 10 librados: watch_flush exit 2024-09-16T10:50:08.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 20 client.4325.objecter shutdown clearing up homeless session... 2024-09-16T10:50:08.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 10 client.4325.objecter successfully canceled tick 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 msgr2=0x7fd950039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 0x7fd950039420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd960008030 comp rx=0 tx=0).stop 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 10 monclient: shutdown 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 msgr2=0x7fd97006fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd968007f20 tx=0x7fd96800d3b0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 shutdown_connections 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9700a38d0 0x7fd97006fae0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd950036f60 0x7fd950039420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9700a25d0 0x7fd97006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 --2- 172.21.15.5:0/399629176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9700a21c0 0x7fd9701cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 >> 172.21.15.5:0/399629176 conn(0x7fd9700a8440 msgr2=0x7fd9700a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 shutdown_connections 2024-09-16T10:50:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 -- 172.21.15.5:0/399629176 wait complete. 2024-09-16T10:50:08.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.400+0000 7fd96f59e700 1 librados: shutdown 2024-09-16T10:50:08.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T10:50:08.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: get_monmap_and_config 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: monmap: 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:08.544101+0000 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:08.544101+0000 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:08.547 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:08.548 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 Processor -- start 2024-09-16T10:50:08.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- start start 2024-09-16T10:50:08.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: init 2024-09-16T10:50:08.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T10:50:08.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-b con 0x7f6cfc099e60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 0x7f6cfc09a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-a con 0x7f6cfc09f1e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6cfc133810 0x7f6cfc133c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-c con 0x7f6cfc133810 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cfc09e3c0 con 0x7f6cfc09f1e0 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cfc09e110 con 0x7f6cfc133810 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc0a0420 con 0x7f6cfc099e60 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient(hunting): authenticate will time out at 867.120442s 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 0x7f6cfc09a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38204/0 (socket says 172.21.15.5:38204) 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 -- 172.21.15.5:0/1391024547 learned_addr learned my addr 172.21.15.5:0/1391024547 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): get_auth_request con 0x7f6cfc099e60 auth_method 0 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): get_auth_request con 0x7f6cfc09f1e0 auth_method 0 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.143:3300/0 conn(0x7f6cfc133810 0x7f6cfc133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): get_auth_request con 0x7f6cfc133810 auth_method 0 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 -- 172.21.15.5:0/1391024547 >> v2:172.21.15.143:3300/0 conn(0x7f6cfc133810 msgr2=0x7f6cfc133c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.143:3300/0 conn(0x7f6cfc133810 0x7f6cfc133c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 -- 172.21.15.5:0/1391024547 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 msgr2=0x7f6cfc09a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 0x7f6cfc09a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 monclient: found mon.noname-b 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 -- 172.21.15.5:0/1391024547 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc09eae0 con 0x7f6cfc099e60 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfaffd700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 0x7f6cfc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/1391024547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc06bac0 tx=0x7f6cec009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=951f7d693849a181 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 1 -- 172.21.15.5:0/1391024547 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011830 con 0x7f6cfc099e60 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: dump: 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.557 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 1 -- 172.21.15.5:0/1391024547 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec011990 con 0x7f6cfc099e60 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 5 monclient: authenticate success, global_id 4328 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:08.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 1 -- 172.21.15.5:0/1391024547 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011d20 con 0x7f6cfc099e60 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 4 set_mon_vals no callback set 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6cfa7fc700 10 monclient: dump: 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.559 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 10 monclient: shutdown 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- 172.21.15.5:0/1391024547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 msgr2=0x7f6cfc09a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/1391024547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc06bac0 tx=0x7f6cec009bf0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- 172.21.15.5:0/1391024547 shutdown_connections 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.143:3300/0 conn(0x7f6cfc133810 0x7f6cfc133c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/1391024547 >> v2:172.21.15.5:3300/0 conn(0x7f6cfc09f1e0 0x7f6cfc09a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/1391024547 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc099e60 0x7f6cfc09a240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- 172.21.15.5:0/1391024547 >> 172.21.15.5:0/1391024547 conn(0x7f6cfc0a4450 msgr2=0x7f6cfc0a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- 172.21.15.5:0/1391024547 shutdown_connections 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.540+0000 7f6d00d6c700 1 -- 172.21.15.5:0/1391024547 wait complete. 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient: monmap: 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:08.561 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:08.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 librados: starting msgr at 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 librados: starting objecter 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 Processor -- start 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 -- start start 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 librados: setting wanted keys 2024-09-16T10:50:08.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 librados: calling monclient init 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient: init 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc099e60 0x7f6cfc1ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-b con 0x7f6cfc099e60 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc09f1e0 0x7f6cfc131e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-a con 0x7f6cfc09f1e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): picked mon.noname-c con 0x7f6cfc133810 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc1caec0 con 0x7f6cfc09f1e0 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc1215f0 con 0x7f6cfc099e60 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc1328e0 con 0x7f6cfc133810 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6d00d6c700 10 monclient(hunting): authenticate will time out at 867.126842s 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc09f1e0 0x7f6cfc131e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38216/0 (socket says 172.21.15.5:38216) 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 -- 172.21.15.5:0/3614333765 learned_addr learned my addr 172.21.15.5:0/3614333765 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): get_auth_request con 0x7f6cfc09f1e0 auth_method 0 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): get_auth_request con 0x7f6cfc133810 auth_method 0 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc099e60 0x7f6cfc1ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): get_auth_request con 0x7f6cfc099e60 auth_method 0 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 -- 172.21.15.5:0/3614333765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc099e60 msgr2=0x7f6cfc1ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc099e60 0x7f6cfc1ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 -- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc09f1e0 msgr2=0x7f6cfc131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc09f1e0 0x7f6cfc131e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 monclient: found mon.noname-c 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.548+0000 7f6cfbfff700 1 -- 172.21.15.5:0/3614333765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40094b0 con 0x7f6cfc133810 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 client.?.objecter ms_handle_connect 0x7f6cfc133810 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cfbfff700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009dd0 tx=0x7f6cf400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4009c40 con 0x7f6cfc133810 2024-09-16T10:50:08.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: dump: 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf400f040 con 0x7f6cfc133810 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400ceb0 con 0x7f6cfc133810 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 5 monclient: authenticate success, global_id 4331 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf9ffb700 4 set_mon_vals no callback set 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 monclient: dump: 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.575 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _renew_subs 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec009570 con 0x7f6cfc133810 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _renew_subs 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc09d1b0 con 0x7f6cfc133810 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 1 librados: init done 2024-09-16T10:50:08.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6cf4014480 con 0x7f6cfc133810 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 --2- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 0x7f6cdc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3562+0+0 (secure 0 0 0) 0x7f6cf4047ea0 con 0x7f6cfc133810 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 client.4331.objecter ms_dispatch 0x7f6cfc000fd0 osd_map(16..16 src has 1..16) v4 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 3 client.4331.objecter handle_osd_map got epochs [16,16] > 0 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 3 client.4331.objecter handle_osd_map decoding full epoch 16 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 20 client.4331.objecter dump_active .. 0 homeless 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc09a240 con 0x7f6cfc133810 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 0x7f6cdc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cfb7fe700 10 monclient: get_auth_request con 0x7f6cdc036f60 auth_method 0 2024-09-16T10:50:08.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cf3fff700 10 client.4331.objecter ms_handle_connect 0x7f6cdc036f60 2024-09-16T10:50:08.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.552+0000 7f6cfb7fe700 1 --2- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 0x7f6cdc039420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6cec000c00 tx=0x7f6cec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.556+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6cf4018020 con 0x7f6cfc133810 2024-09-16T10:50:08.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.556+0000 7f6cf3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.556+0000 7f6cf3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6d00d6c700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:50:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6d00d6c700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:50:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6d00d6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6cfc121730 con 0x7f6cfc133810 2024-09-16T10:50:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6cf3fff700 1 -- 172.21.15.5:0/3614333765 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6cf4011680 con 0x7f6cfc133810 2024-09-16T10:50:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6cf3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T10:50:08.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.728+0000 7f6cf3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:08.734 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 8 2024-09-16T10:50:08.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 10 librados: watch_flush enter 2024-09-16T10:50:08.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 10 librados: watch_flush exit 2024-09-16T10:50:08.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-16T10:50:08.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 10 client.4331.objecter successfully canceled tick 2024-09-16T10:50:08.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 msgr2=0x7f6cdc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 0x7f6cdc039420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6cec000c00 tx=0x7f6cec008030 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 10 monclient: shutdown 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 msgr2=0x7f6cfc1323a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009dd0 tx=0x7f6cf400c3a0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 shutdown_connections 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6cfc133810 0x7f6cfc1323a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6cdc036f60 0x7f6cdc039420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6cfc09f1e0 0x7f6cfc131e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 --2- 172.21.15.5:0/3614333765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cfc099e60 0x7f6cfc1ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 >> 172.21.15.5:0/3614333765 conn(0x7f6cfc0a4450 msgr2=0x7f6cfc0a1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 shutdown_connections 2024-09-16T10:50:08.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 -- 172.21.15.5:0/3614333765 wait complete. 2024-09-16T10:50:08.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.732+0000 7f6d00d6c700 1 librados: shutdown 2024-09-16T10:50:08.749 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T10:50:08.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:50:08.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: get_monmap_and_config 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: monmap: 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:08.857421+0000 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:08.857421+0000 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:08.860 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:08.861 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 Processor -- start 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 -- start start 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: init 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4adc0a0900 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4adc0a13f0 0x7f4adc09abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-c con 0x7f4adc0a13f0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-a con 0x7f4adc13aba0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc073c20 con 0x7f4adc0a0900 2024-09-16T10:50:08.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc071d60 con 0x7f4adc13aba0 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc09a1b0 con 0x7f4adc0a13f0 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient(hunting): authenticate will time out at 867.433780s 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49114/0 (socket says 172.21.15.5:49114) 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 1 -- 172.21.15.5:0/4258748423 learned_addr learned my addr 172.21.15.5:0/4258748423 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): get_auth_request con 0x7f4adc13aba0 auth_method 0 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/4258748423 >> v2:172.21.15.143:3300/0 conn(0x7f4adc0a13f0 0x7f4adc09abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): get_auth_request con 0x7f4adc0a13f0 auth_method 0 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 -- 172.21.15.5:0/4258748423 >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 msgr2=0x7f4adc13cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/4258748423 >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 -- 172.21.15.5:0/4258748423 >> v1:172.21.15.143:6789/0 conn(0x7f4adc0a0900 legacy=0x7f4adc0a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 monclient: found mon.noname-c 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 -- 172.21.15.5:0/4258748423 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc09a750 con 0x7f4adc0a13f0 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae0c51700 1 --2- 172.21.15.5:0/4258748423 >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:08.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/4258748423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc0a13f0 0x7f4adc09abe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04e720 tx=0x7f4acc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d95ff48cf0323d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 1 -- 172.21.15.5:0/4258748423 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00e070 con 0x7f4adc0a13f0 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: dump: 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 5 monclient: authenticate success, global_id 4390 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 1 -- 172.21.15.5:0/4258748423 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc00bb20 con 0x7f4adc0a13f0 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 1 -- 172.21.15.5:0/4258748423 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc012430 con 0x7f4adc0a13f0 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4ae1452700 4 set_mon_vals no callback set 2024-09-16T10:50:08.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.856+0000 7f4adaffd700 10 monclient: dump: 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: shutdown 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- 172.21.15.5:0/4258748423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc0a13f0 msgr2=0x7f4adc09abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- 172.21.15.5:0/4258748423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc0a13f0 0x7f4adc09abe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04e720 tx=0x7f4acc00b330 comp rx=0 tx=0).stop 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- 172.21.15.5:0/4258748423 shutdown_connections 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- 172.21.15.5:0/4258748423 >> v2:172.21.15.5:3300/0 conn(0x7f4adc13aba0 0x7f4adc13cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- 172.21.15.5:0/4258748423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc0a13f0 0x7f4adc09abe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- 172.21.15.5:0/4258748423 >> 172.21.15.5:0/4258748423 conn(0x7f4adc06a4f0 msgr2=0x7f4adc06a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- 172.21.15.5:0/4258748423 shutdown_connections 2024-09-16T10:50:08.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- 172.21.15.5:0/4258748423 wait complete. 2024-09-16T10:50:08.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: build_initial_monmap 2024-09-16T10:50:08.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: monmap: 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 librados: starting msgr at 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 librados: starting objecter 2024-09-16T10:50:08.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 Processor -- start 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- start start 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 librados: setting wanted keys 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 librados: calling monclient init 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: init 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-a con 0x7f4adc0a0900 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-c con 0x7f4adc0a13f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc13aba0 0x7f4adc1ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): picked mon.noname-b con 0x7f4adc13aba0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:08.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc120450 con 0x7f4adc0a0900 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc13d130 con 0x7f4adc13aba0 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc120910 con 0x7f4adc0a13f0 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae1452700 10 monclient(hunting): authenticate will time out at 867.438768s 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49122/0 (socket says 172.21.15.5:49122) 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38230/0 (socket says 172.21.15.5:38230) 2024-09-16T10:50:08.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 1 -- 172.21.15.5:0/3061821225 learned_addr learned my addr 172.21.15.5:0/3061821225 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): get_auth_request con 0x7f4adc0a0900 auth_method 0 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): get_auth_request con 0x7f4adc0a13f0 auth_method 0 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc13aba0 0x7f4adc1ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): get_auth_request con 0x7f4adc13aba0 auth_method 0 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ae0c51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 -- 172.21.15.5:0/3061821225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc13aba0 msgr2=0x7f4adc1ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc13aba0 0x7f4adc1ca940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 -- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 msgr2=0x7f4adc1c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 monclient: found mon.noname-c 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 -- 172.21.15.5:0/3061821225 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4009570 con 0x7f4adc0a13f0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adbfff700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4adb7fe700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4acc016000 tx=0x7f4acc004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ad8ff9700 10 client.?.objecter ms_handle_connect 0x7f4adc0a13f0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.860+0000 7f4ad8ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc012430 con 0x7f4adc0a13f0 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: dump: 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: _finish_auth 0 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:50:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 5 monclient: authenticate success, global_id 4334 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc020070 con 0x7f4adc0a13f0 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _renew_subs 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4acc009a70 con 0x7f4adc0a13f0 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc004280 con 0x7f4adc0a13f0 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 monclient: dump: 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _renew_subs 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc09c130 con 0x7f4adc0a13f0 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 1 librados: init done 2024-09-16T10:50:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ada7fc700 4 set_mon_vals no callback set 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4acc0043e0 con 0x7f4adc0a13f0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc120590 con 0x7f4adc0a13f0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 --2- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 0x7f4ac8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f4acc022070 con 0x7f4adc0a13f0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 client.4334.objecter ms_dispatch 0x7f4adc000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 3 client.4334.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 3 client.4334.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 20 client.4334.objecter dump_active .. 0 homeless 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4adbfff700 1 --2- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 0x7f4ac8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4adbfff700 10 monclient: get_auth_request con 0x7f4ac8036f60 auth_method 0 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4ad8ff9700 10 client.4334.objecter ms_handle_connect 0x7f4ac8036f60 2024-09-16T10:50:08.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.864+0000 7f4adbfff700 1 --2- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 0x7f4ac8039420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0001ce0 tx=0x7f4ad0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.872+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4adc120590 con 0x7f4adc0a13f0 2024-09-16T10:50:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.872+0000 7f4ad8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:08.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:08.872+0000 7f4ad8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:09.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.040+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4adc13cfc0 con 0x7f4ac8036f60 2024-09-16T10:50:09.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.044+0000 7f4ad8ff9700 1 -- 172.21.15.5:0/3061821225 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40948 (secure 0 0 0) 0x7f4adc13cfc0 con 0x7f4ac8036f60 2024-09-16T10:50:09.050 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:09.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 10 librados: watch_flush enter 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 10 librados: watch_flush exit 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 20 client.4334.objecter shutdown clearing up homeless session... 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 10 client.4334.objecter successfully canceled tick 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 msgr2=0x7f4ac8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 0x7f4ac8039420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0001ce0 tx=0x7f4ad0009040 comp rx=0 tx=0).stop 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 10 monclient: shutdown 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 msgr2=0x7f4adc135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4acc016000 tx=0x7f4acc004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 shutdown_connections 2024-09-16T10:50:09.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4adc13aba0 0x7f4adc1ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4adc0a13f0 0x7f4adc135f80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4ac8036f60 0x7f4ac8039420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 --2- 172.21.15.5:0/3061821225 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4adc0a0900 0x7f4adc1c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 >> 172.21.15.5:0/3061821225 conn(0x7f4adc06a4f0 msgr2=0x7f4adc09e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 shutdown_connections 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 -- 172.21.15.5:0/3061821225 wait complete. 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.048+0000 7f4ae1452700 1 librados: shutdown 2024-09-16T10:50:09.053 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T10:50:09.064 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T10:50:08.575371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168092,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566062948,"statfs":{"total":579820584960,"available":579648458752,"internally_reserved":0,"allocated":1900544,"data_stored":1308688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.285947"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526459+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T10:50:03.526459+0000","last_peered":"2024-09-16T10:50:03.526459+0000","last_clean":"2024-09-16T10:50:03.526459+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T10:50:03.526459+0000","last_undegraded":"2024-09-16T10:50:03.526459+0000","last_fullsized":"2024-09-16T10:50:03.526459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429266+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T10:50:03.429266+0000","last_peered":"2024-09-16T10:50:03.429266+0000","last_clean":"2024-09-16T10:50:03.429266+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T10:50:03.429266+0000","last_undegraded":"2024-09-16T10:50:03.429266+0000","last_fullsized":"2024-09-16T10:50:03.429266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526540+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T10:50:03.526540+0000","last_peered":"2024-09-16T10:50:03.526540+0000","last_clean":"2024-09-16T10:50:03.526540+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T10:50:03.526540+0000","last_undegraded":"2024-09-16T10:50:03.526540+0000","last_fullsized":"2024-09-16T10:50:03.526540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429452+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T10:50:03.429452+0000","last_peered":"2024-09-16T10:50:03.429452+0000","last_clean":"2024-09-16T10:50:03.429452+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T10:50:03.429452+0000","last_undegraded":"2024-09-16T10:50:03.429452+0000","last_fullsized":"2024-09-16T10:50:03.429452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.430870+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T10:50:03.430870+0000","last_peered":"2024-09-16T10:50:03.430870+0000","last_clean":"2024-09-16T10:50:03.430870+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T10:50:03.430870+0000","last_undegraded":"2024-09-16T10:50:03.430870+0000","last_fullsized":"2024-09-16T10:50:03.430870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.688026+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T10:50:03.688026+0000","last_peered":"2024-09-16T10:50:03.688026+0000","last_clean":"2024-09-16T10:50:03.688026+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T10:50:03.688026+0000","last_undegraded":"2024-09-16T10:50:03.688026+0000","last_fullsized":"2024-09-16T10:50:03.688026+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526951+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T10:50:03.526951+0000","last_peered":"2024-09-16T10:50:03.526951+0000","last_clean":"2024-09-16T10:50:03.526951+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T10:50:03.526951+0000","last_undegraded":"2024-09-16T10:50:03.526951+0000","last_fullsized":"2024-09-16T10:50:03.526951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.428485+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T10:50:03.428485+0000","last_peered":"2024-09-16T10:50:03.428485+0000","last_clean":"2024-09-16T10:50:03.428485+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T10:50:03.428485+0000","last_undegraded":"2024-09-16T10:50:03.428485+0000","last_fullsized":"2024-09-16T10:50:03.428485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.525806+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T10:50:03.525806+0000","last_peered":"2024-09-16T10:50:03.525806+0000","last_clean":"2024-09-16T10:50:03.525806+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T10:50:03.525806+0000","last_undegraded":"2024-09-16T10:50:03.525806+0000","last_fullsized":"2024-09-16T10:50:03.525806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":42487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":131072,"data_stored":38359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:50:09.065 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:50:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: get_monmap_and_config 2024-09-16T10:50:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: monmap: 2024-09-16T10:50:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.175 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:09.173160+0000 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:09.173160+0000 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:09.176 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 Processor -- start 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 -- start start 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: init 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb48070750 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb480a04b0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb4813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-b con 0x7fcb480a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4809fb60 con 0x7fcb480a04b0 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480a0090 con 0x7fcb48070750 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4809f7f0 con 0x7fcb480a14c0 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient(hunting): authenticate will time out at 867.749515s 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fcb480a04b0 0x7fcb4813ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59108/0 (socket says 172.21.15.5:59108) 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb4813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 1 -- 172.21.15.5:0/4160291070 learned_addr learned my addr 172.21.15.5:0/4160291070 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): get_auth_request con 0x7fcb480a14c0 auth_method 0 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461562023 0 0) 0x7fcb4809fb60 con 0x7fcb480a04b0 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _negotiate 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): my global_id is 4413 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb340035d0 con 0x7fcb480a04b0 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700540477 0 0) 0x7fcb480a0090 con 0x7fcb48070750 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _negotiate 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): my global_id is 4399 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb4809fb60 con 0x7fcb48070750 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436332277 0 0) 0x7fcb340035d0 con 0x7fcb480a04b0 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 msgr2=0x7fcb4813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb4813e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 >> v1:172.21.15.143:6789/0 conn(0x7fcb48070750 legacy=0x7fcb48070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4809ad40 con 0x7fcb480a04b0 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7fcb30001670 con 0x7fcb480a04b0 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 5 monclient: authenticate success, global_id 4413 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: dump: 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.186 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 >> v1:172.21.15.5:6789/0 conn(0x7fcb480a04b0 legacy=0x7fcb4813ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb34003be0 0x7fcb34003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): picked mon.b con 0x7fcb34003be0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb34004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): picked mon.c con 0x7fcb480a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb34004db0 0x7fcb34005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): picked mon.a con 0x7fcb34004db0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:09.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4809f7f0 con 0x7fcb34004db0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3400bbf0 con 0x7fcb34003be0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3400bdb0 con 0x7fcb480a14c0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb34004db0 0x7fcb34005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): get_auth_request con 0x7fcb34004db0 auth_method 0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb34004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb34003be0 0x7fcb34003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 10 monclient(hunting): get_auth_request con 0x7fcb34003be0 auth_method 0 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 10 monclient(hunting): get_auth_request con 0x7fcb480a14c0 auth_method 0 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb46ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:09.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient(hunting): handle_auth_done global_id 4413 payload 293 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 -- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 msgr2=0x7fcb34004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb480a14c0 0x7fcb34004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 -- 172.21.15.5:0/4160291070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb34003be0 msgr2=0x7fcb34003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb34003be0 0x7fcb34003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 monclient: found mon.a 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 -- 172.21.15.5:0/4160291070 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340035d0 con 0x7fcb34004db0 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb477fe700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb34003be0 0x7fcb34003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb47fff700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb34004db0 0x7fcb34005230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009f90 tx=0x7fcb340058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5dec298100d881 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3800ec30 con 0x7fcb34004db0 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: dump: 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.191 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3800f040 con 0x7fcb34004db0 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb4ca73700 4 set_mon_vals no callback set 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 1 -- 172.21.15.5:0/4160291070 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38014480 con 0x7fcb34004db0 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.172+0000 7fcb467fc700 10 monclient: dump: 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: shutdown 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb34004db0 msgr2=0x7fcb34005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/4160291070 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb34004db0 0x7fcb34005230 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009f90 tx=0x7fcb340058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 1 -- 172.21.15.5:0/4160291070 reap_dead start 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/4160291070 shutdown_connections 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/4160291070 >> 172.21.15.5:0/4160291070 conn(0x7fcb480a45d0 msgr2=0x7fcb480a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/4160291070 shutdown_connections 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/4160291070 wait complete. 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: monmap: 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 librados: starting msgr at 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 librados: starting objecter 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 Processor -- start 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- start start 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 librados: setting wanted keys 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 librados: calling monclient init 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: init 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb34004db0 0x7fcb481c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-c con 0x7fcb34004db0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-a con 0x7fcb4812fcb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb481ca810 0x7fcb481cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): picked mon.noname-b con 0x7fcb481ca810 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4811adb0 con 0x7fcb4812fcb0 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4809bc40 con 0x7fcb481ca810 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4809b0d0 con 0x7fcb34004db0 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient(hunting): authenticate will time out at 867.754505s 2024-09-16T10:50:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49142/0 (socket says 172.21.15.5:49142) 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 -- 172.21.15.5:0/751722886 learned_addr learned my addr 172.21.15.5:0/751722886 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): get_auth_request con 0x7fcb4812fcb0 auth_method 0 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb34004db0 0x7fcb481c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): get_auth_request con 0x7fcb34004db0 auth_method 0 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb481ca810 0x7fcb481cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 10 monclient(hunting): get_auth_request con 0x7fcb481ca810 auth_method 0 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 -- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb34004db0 msgr2=0x7fcb481c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb34004db0 0x7fcb481c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 -- 172.21.15.5:0/751722886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb481ca810 msgr2=0x7fcb481cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb481ca810 0x7fcb481cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 monclient: found mon.noname-a 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 -- 172.21.15.5:0/751722886 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4813ba30 con 0x7fcb4812fcb0 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb46ffd700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000e7b0 tx=0x7fcb3000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 client.?.objecter ms_handle_connect 0x7fcb4812fcb0 2024-09-16T10:50:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:09.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb30012590 con 0x7fcb4812fcb0 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: dump: 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.204 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 5 monclient: authenticate success, global_id 4416 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb30012710 con 0x7fcb4812fcb0 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _renew_subs 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48130670 con 0x7fcb4812fcb0 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb30012ac0 con 0x7fcb4812fcb0 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb457fa700 4 set_mon_vals no callback set 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb3ffff700 10 monclient: dump: 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _renew_subs 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4813ac40 con 0x7fcb4812fcb0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.176+0000 7fcb4ca73700 1 librados: init done 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fcb30012c20 con 0x7fcb4812fcb0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb4ca73700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb4ca73700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb4ca73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb48070b30 con 0x7fcb4812fcb0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 1 --2- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 0x7fcb28039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7fcb30047460 con 0x7fcb4812fcb0 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 10 client.4416.objecter ms_dispatch 0x7fcb48000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:09.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 3 client.4416.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 3 client.4416.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 20 client.4416.objecter dump_active .. 0 homeless 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb477fe700 1 --2- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 0x7fcb28039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb477fe700 10 monclient: get_auth_request con 0x7fcb28036f90 auth_method 0 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb3ffff700 10 client.4416.objecter ms_handle_connect 0x7fcb28036f90 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.180+0000 7fcb477fe700 1 --2- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 0x7fcb28039450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb40007fd0 tx=0x7fcb40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.188+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcb30012ed0 con 0x7fcb4812fcb0 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.188+0000 7fcb3ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.188+0000 7fcb3ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:09.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.356+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb4809ca90 con 0x7fcb28036f90 2024-09-16T10:50:09.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb3ffff700 1 -- 172.21.15.5:0/751722886 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40948 (secure 0 0 0) 0x7fcb4809ca90 con 0x7fcb28036f90 2024-09-16T10:50:09.365 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 10 librados: watch_flush enter 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 10 librados: watch_flush exit 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 20 client.4416.objecter shutdown clearing up homeless session... 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 10 client.4416.objecter successfully canceled tick 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 msgr2=0x7fcb28039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 0x7fcb28039450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcb40007fd0 tx=0x7fcb40009040 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 10 monclient: shutdown 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 msgr2=0x7fcb48130130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000e7b0 tx=0x7fcb3000eb70 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 shutdown_connections 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb481ca810 0x7fcb481cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> v2:172.21.15.143:6812/1098205034 conn(0x7fcb28036f90 0x7fcb28039450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb4812fcb0 0x7fcb48130130 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 --2- 172.21.15.5:0/751722886 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fcb34004db0 0x7fcb481c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 >> 172.21.15.5:0/751722886 conn(0x7fcb480a45d0 msgr2=0x7fcb480a2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.360+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 shutdown_connections 2024-09-16T10:50:09.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.364+0000 7fcb4ca73700 1 -- 172.21.15.5:0/751722886 wait complete. 2024-09-16T10:50:09.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.364+0000 7fcb4ca73700 1 librados: shutdown 2024-09-16T10:50:09.369 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T10:50:09.380 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T10:50:09.290582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168092,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":166134,"kb_avail":566062948,"statfs":{"total":579820584960,"available":579648458752,"internally_reserved":0,"allocated":1900544,"data_stored":1308688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":170121894},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001159"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526459+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T10:50:03.526459+0000","last_peered":"2024-09-16T10:50:03.526459+0000","last_clean":"2024-09-16T10:50:03.526459+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T10:50:03.526459+0000","last_undegraded":"2024-09-16T10:50:03.526459+0000","last_fullsized":"2024-09-16T10:50:03.526459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429266+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T10:50:03.429266+0000","last_peered":"2024-09-16T10:50:03.429266+0000","last_clean":"2024-09-16T10:50:03.429266+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T10:50:03.429266+0000","last_undegraded":"2024-09-16T10:50:03.429266+0000","last_fullsized":"2024-09-16T10:50:03.429266+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526540+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T10:50:03.526540+0000","last_peered":"2024-09-16T10:50:03.526540+0000","last_clean":"2024-09-16T10:50:03.526540+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T10:50:03.526540+0000","last_undegraded":"2024-09-16T10:50:03.526540+0000","last_fullsized":"2024-09-16T10:50:03.526540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.429452+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T10:50:03.429452+0000","last_peered":"2024-09-16T10:50:03.429452+0000","last_clean":"2024-09-16T10:50:03.429452+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T10:50:03.429452+0000","last_undegraded":"2024-09-16T10:50:03.429452+0000","last_fullsized":"2024-09-16T10:50:03.429452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.430870+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T10:50:03.430870+0000","last_peered":"2024-09-16T10:50:03.430870+0000","last_clean":"2024-09-16T10:50:03.430870+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T10:50:03.430870+0000","last_undegraded":"2024-09-16T10:50:03.430870+0000","last_fullsized":"2024-09-16T10:50:03.430870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.688026+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T10:50:03.688026+0000","last_peered":"2024-09-16T10:50:03.688026+0000","last_clean":"2024-09-16T10:50:03.688026+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T10:50:03.688026+0000","last_undegraded":"2024-09-16T10:50:03.688026+0000","last_fullsized":"2024-09-16T10:50:03.688026+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.526951+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T10:50:03.526951+0000","last_peered":"2024-09-16T10:50:03.526951+0000","last_clean":"2024-09-16T10:50:03.526951+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T10:50:03.526951+0000","last_undegraded":"2024-09-16T10:50:03.526951+0000","last_fullsized":"2024-09-16T10:50:03.526951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.428485+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T10:50:03.428485+0000","last_peered":"2024-09-16T10:50:03.428485+0000","last_clean":"2024-09-16T10:50:03.428485+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T10:50:03.428485+0000","last_undegraded":"2024-09-16T10:50:03.428485+0000","last_fullsized":"2024-09-16T10:50:03.428485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T10:50:03.525806+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T10:50:03.525806+0000","last_peered":"2024-09-16T10:50:03.525806+0000","last_clean":"2024-09-16T10:50:03.525806+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T10:50:03.525806+0000","last_undegraded":"2024-09-16T10:50:03.525806+0000","last_fullsized":"2024-09-16T10:50:03.525806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":42487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":131072,"data_stored":38359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":571415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:50:09.381 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T10:50:09.381 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T10:50:09.381 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T10:50:09.394 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T10:50:09.394 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T10:50:09.395 DEBUG:teuthology.orchestra.run.smithi143:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T10:50:09.395 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T10:50:09.395 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726483809, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T10:50:09.395 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:50:09.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:50:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:50:09.403 INFO:teuthology.orchestra.run.smithi143.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T10:50:09.403 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T10:50:09.451 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T10:50:09.451 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T10:50:09.458 INFO:tasks.workunit:timeout=3h 2024-09-16T10:50:09.458 INFO:tasks.workunit:cleanup=True 2024-09-16T10:50:09.459 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T10:50:09.467 INFO:tasks.workunit.client.0.smithi143.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T10:50:09.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: get_monmap_and_config 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: monmap: 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:09.511447+0000 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:09.511447+0000 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:09.514 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:09.515 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 Processor -- start 2024-09-16T10:50:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 -- start start 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: init 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb280a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb2809f780 0x7ffb2813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-c con 0x7ffb2809f780 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffb280a36e0 0x7ffb2813d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-a con 0x7ffb280a36e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb280a3c40 con 0x7ffb280a2bf0 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb280a4300 con 0x7ffb280a36e0 2024-09-16T10:50:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2809fda0 con 0x7ffb2809f780 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.508+0000 7ffb2d63c700 10 monclient(hunting): authenticate will time out at 868.087862s 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffb280a36e0 0x7ffb2813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffb280a36e0 0x7ffb2813d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49150/0 (socket says 172.21.15.5:49150) 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 -- 172.21.15.5:0/1235751492 learned_addr learned my addr 172.21.15.5:0/1235751492 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): get_auth_request con 0x7ffb280a36e0 auth_method 0 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 1 --2- 172.21.15.5:0/1235751492 >> v2:172.21.15.143:3300/0 conn(0x7ffb2809f780 0x7ffb2813ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): get_auth_request con 0x7ffb2809f780 auth_method 0 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 -- 172.21.15.5:0/1235751492 >> v2:172.21.15.143:3300/0 conn(0x7ffb2809f780 msgr2=0x7ffb2813ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 --2- 172.21.15.5:0/1235751492 >> v2:172.21.15.143:3300/0 conn(0x7ffb2809f780 0x7ffb2813ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 -- 172.21.15.5:0/1235751492 >> v1:172.21.15.143:6789/0 conn(0x7ffb280a2bf0 legacy=0x7ffb280a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 monclient: found mon.noname-a 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 -- 172.21.15.5:0/1235751492 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb280a0080 con 0x7ffb280a36e0 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2ce3b700 1 --2- 172.21.15.5:0/1235751492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813d350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb280a60a0 tx=0x7ffb2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830fa125e1fd1751 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 1 -- 172.21.15.5:0/1235751492 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000e070 con 0x7ffb280a36e0 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: dump: 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 5 monclient: authenticate success, global_id 4419 2024-09-16T10:50:09.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 1 -- 172.21.15.5:0/1235751492 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb20003a40 con 0x7ffb280a36e0 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 4 set_mon_vals no callback set 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 1 -- 172.21.15.5:0/1235751492 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20003df0 con 0x7ffb280a36e0 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb26ffd700 10 monclient: dump: 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 10 monclient: shutdown 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1235751492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 msgr2=0x7ffb2813d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1235751492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813d350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb280a60a0 tx=0x7ffb2000b330 comp rx=0 tx=0).stop 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1235751492 shutdown_connections 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1235751492 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813d350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1235751492 >> v2:172.21.15.143:3300/0 conn(0x7ffb2809f780 0x7ffb2813ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1235751492 >> 172.21.15.5:0/1235751492 conn(0x7ffb280a53e0 msgr2=0x7ffb28073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1235751492 shutdown_connections 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1235751492 wait complete. 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 10 monclient: monmap: 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:09.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 librados: starting msgr at 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 librados: starting objecter 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 Processor -- start 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 -- start start 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 librados: setting wanted keys 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 1 librados: calling monclient init 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.512+0000 7ffb2d63c700 10 monclient: init 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2809f780 0x7ffb281354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-b con 0x7ffb2809f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:09.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-c con 0x7ffb280a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): picked mon.noname-a con 0x7ffb280a36e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2811fb50 con 0x7ffb280a36e0 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2809f4e0 con 0x7ffb2809f780 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2813cff0 con 0x7ffb280a2bf0 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient(hunting): authenticate will time out at 868.092774s 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38274/0 (socket says 172.21.15.5:38274) 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 -- 172.21.15.5:0/1475704625 learned_addr learned my addr 172.21.15.5:0/1475704625 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): get_auth_request con 0x7ffb280a2bf0 auth_method 0 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): get_auth_request con 0x7ffb280a36e0 auth_method 0 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2809f780 0x7ffb281354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2ce3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): get_auth_request con 0x7ffb2809f780 auth_method 0 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2024-09-16T10:50:09.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 -- 172.21.15.5:0/1475704625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2809f780 msgr2=0x7ffb281354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2809f780 0x7ffb281354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 -- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 msgr2=0x7ffb2813aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 monclient: found mon.noname-c 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 -- 172.21.15.5:0/1475704625 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20009b30 con 0x7ffb280a2bf0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb277fe700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800e980 tx=0x7ffb1800ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 client.?.objecter ms_handle_connect 0x7ffb280a2bf0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb180096e0 con 0x7ffb280a2bf0 2024-09-16T10:50:09.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: dump: 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 5 monclient: authenticate success, global_id 4343 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1800c690 con 0x7ffb280a2bf0 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _renew_subs 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1800ca20 con 0x7ffb280a2bf0 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb280a78b0 con 0x7ffb280a2bf0 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb267fc700 4 set_mon_vals no callback set 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:09.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 10 monclient: dump: 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _renew_subs 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2813c690 con 0x7ffb280a2bf0 2024-09-16T10:50:09.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 librados: init done 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb280a2fd0 con 0x7ffb280a2bf0 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.516+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ffb1800cb80 con 0x7ffb280a2bf0 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 1 --2- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 0x7ffb14039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7ffb18047700 con 0x7ffb280a2bf0 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 10 client.4343.objecter ms_dispatch 0x7ffb28000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 3 client.4343.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:09.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 3 client.4343.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 20 client.4343.objecter dump_active .. 0 homeless 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb27fff700 1 --2- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 0x7ffb14039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb27fff700 10 monclient: get_auth_request con 0x7ffb14036f60 auth_method 0 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb27fff700 1 --2- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 0x7ffb14039420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c005150 tx=0x7ffb1c007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.520+0000 7ffb24ff9700 10 client.4343.objecter ms_handle_connect 0x7ffb14036f60 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.528+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffb1801bd80 con 0x7ffb280a2bf0 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.528+0000 7ffb24ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.528+0000 7ffb24ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:09.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb2d63c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:50:09.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb2d63c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:09.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb2d63c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffb280721d0 con 0x7ffb280a2bf0 2024-09-16T10:50:09.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb24ff9700 1 -- 172.21.15.5:0/1475704625 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7ffb18017070 con 0x7ffb280a2bf0 2024-09-16T10:50:09.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb24ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:09.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb24ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:09.769 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.764+0000 7ffb2d63c700 10 librados: watch_flush enter 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 10 librados: watch_flush exit 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 20 client.4343.objecter shutdown clearing up homeless session... 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 10 client.4343.objecter successfully canceled tick 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 msgr2=0x7ffb14039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 0x7ffb14039420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c005150 tx=0x7ffb1c007ee0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 10 monclient: shutdown 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 msgr2=0x7ffb281359e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800e980 tx=0x7ffb1800ec90 comp rx=0 tx=0).stop 2024-09-16T10:50:09.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 shutdown_connections 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffb14036f60 0x7ffb14039420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffb280a36e0 0x7ffb2813aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffb280a2bf0 0x7ffb281359e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 --2- 172.21.15.5:0/1475704625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2809f780 0x7ffb281354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 >> 172.21.15.5:0/1475704625 conn(0x7ffb280a53e0 msgr2=0x7ffb280a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 shutdown_connections 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 -- 172.21.15.5:0/1475704625 wait complete. 2024-09-16T10:50:09.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.768+0000 7ffb2d63c700 1 librados: shutdown 2024-09-16T10:50:09.784 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:50:09.784 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:50:09.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:50:09.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:50:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: get_monmap_and_config 2024-09-16T10:50:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: monmap: 2024-09-16T10:50:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:09.898129+0000 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:09.898129+0000 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:09.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 Processor -- start 2024-09-16T10:50:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 -- start start 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: init 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4b00a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 0x7fb4b013ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b00a3770 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4b013b1c0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00a3db0 con 0x7fb4b013b1c0 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00a4b70 con 0x7fb4b00a2c80 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00a3b50 con 0x7fb4b00a3770 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b74fb700 10 monclient(hunting): authenticate will time out at 868.474644s 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b6cfa700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb4b013b1c0 0x7fb4b013d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59110/0 (socket says 172.21.15.5:59110) 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b6cfa700 1 -- 172.21.15.5:0/2995008495 learned_addr learned my addr 172.21.15.5:0/2995008495 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 0x7fb4b013ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request con 0x7fb4b00a3770 auth_method 0 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 -- 172.21.15.5:0/2995008495 >> v1:172.21.15.143:6789/0 conn(0x7fb4b00a2c80 legacy=0x7fb4b00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 -- 172.21.15.5:0/2995008495 >> v1:172.21.15.5:6789/0 conn(0x7fb4b013b1c0 legacy=0x7fb4b013d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 monclient: found mon.noname-b 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b54f7700 1 -- 172.21.15.5:0/2995008495 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999028389 0 0) 0x7fb4b00a3db0 con 0x7fb4b013b1c0 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 -- 172.21.15.5:0/2995008495 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00a0850 con 0x7fb4b00a3770 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b54f7700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.896+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/2995008495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 0x7fb4b013ac80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009dc0 tx=0x7fb4ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0549b1a43ed7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 1 -- 172.21.15.5:0/2995008495 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00e070 con 0x7fb4b00a3770 2024-09-16T10:50:09.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: dump: 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 5 monclient: authenticate success, global_id 4346 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 1 -- 172.21.15.5:0/2995008495 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac00bbd0 con 0x7fb4b00a3770 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 1 -- 172.21.15.5:0/2995008495 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac012430 con 0x7fb4b00a3770 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b54f7700 10 monclient: dump: 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 4 set_mon_vals no callback set 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 10 monclient: shutdown 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 -- 172.21.15.5:0/2995008495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 msgr2=0x7fb4b013ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/2995008495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 0x7fb4b013ac80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009dc0 tx=0x7fb4ac00b330 comp rx=0 tx=0).stop 2024-09-16T10:50:09.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 -- 172.21.15.5:0/2995008495 shutdown_connections 2024-09-16T10:50:09.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/2995008495 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a3770 0x7fb4b013ac80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 -- 172.21.15.5:0/2995008495 >> 172.21.15.5:0/2995008495 conn(0x7fb4b00a4d80 msgr2=0x7fb4b00a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 -- 172.21.15.5:0/2995008495 shutdown_connections 2024-09-16T10:50:09.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 -- 172.21.15.5:0/2995008495 wait complete. 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 10 monclient: build_initial_monmap 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 10 monclient: monmap: 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 librados: starting msgr at 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 librados: starting objecter 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.900+0000 7fb4b74fb700 1 Processor -- start 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- start start 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 librados: setting wanted keys 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 librados: calling monclient init 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: init 2024-09-16T10:50:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-c con 0x7fb4b00a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b00a3770 0x7fb4b01382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-a con 0x7fb4b00a3770 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b013b1c0 0x7fb4b01208c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b013b1c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01241f0 con 0x7fb4b00a3770 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0124440 con 0x7fb4b013b1c0 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b00a7670 con 0x7fb4b00a2c80 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient(hunting): authenticate will time out at 868.481495s 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38294/0 (socket says 172.21.15.5:38294) 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b00a3770 0x7fb4b01382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 -- 172.21.15.5:0/1614159111 learned_addr learned my addr 172.21.15.5:0/1614159111 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): get_auth_request con 0x7fb4b00a2c80 auth_method 0 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request con 0x7fb4b00a3770 auth_method 0 2024-09-16T10:50:09.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b013b1c0 0x7fb4b01208c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): get_auth_request con 0x7fb4b013b1c0 auth_method 0 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b6cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b013b1c0 msgr2=0x7fb4b01208c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4b013b1c0 0x7fb4b01208c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b00a3770 msgr2=0x7fb4b01382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b00a3770 0x7fb4b01382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient: _finish_hunting 0 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 monclient: found mon.noname-c 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:09.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac009a70 con 0x7fb4b00a2c80 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b00a3770 0x7fb4b01382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 client.?.objecter ms_handle_connect 0x7fb4b00a2c80 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b64f9700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000c530 tx=0x7fb4a000c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00052d0 con 0x7fb4b00a2c80 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: dump: 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.922 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 5 monclient: authenticate success, global_id 4349 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _renew_subs 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a0005430 con 0x7fb4b00a2c80 2024-09-16T10:50:09.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b00a7120 con 0x7fb4b00a2c80 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00055a0 con 0x7fb4b00a2c80 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b4cf6700 4 set_mon_vals no callback set 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4a6ffd700 10 monclient: dump: 2024-09-16T10:50:09.924 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _renew_subs 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b0122d10 con 0x7fb4b00a2c80 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.904+0000 7fb4b74fb700 1 librados: init done 2024-09-16T10:50:09.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b74fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 0x7fb4b002b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01245d0 con 0x7fb4b01215d0 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 0x7fb4b002b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb4a00038e0 con 0x7fb4b00a2c80 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient: get_auth_request con 0x7fb4b01215d0 auth_method 0 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 --2- 172.21.15.5:0/1614159111 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb494036f60 0x7fb494039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_done global_id 4431 payload 824 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7fb4a0046a50 con 0x7fb4b00a2c80 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 client.4349.objecter ms_dispatch 0x7fb4b0000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 3 client.4349.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 3 client.4349.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4b00a4400 con 0x7fb4b01215d0 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 20 client.4349.objecter dump_active .. 0 homeless 2024-09-16T10:50:09.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 0x7fb4b002b6a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009dc0 tx=0x7fb4ac004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 client.4349.objecter ms_handle_connect 0x7fb4b01215d0 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 client.4349.objecter resend_mon_ops 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b6cfa700 1 --2- 172.21.15.5:0/1614159111 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb494036f60 0x7fb494039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b6cfa700 10 monclient: get_auth_request con 0x7fb494036f60 auth_method 0 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac012e50 con 0x7fb4b01215d0 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 monclient: dump: 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.928 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fb4ac01caa0 con 0x7fb4b01215d0 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 msgr2=0x7fb4b002b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 0x7fb4b002b6a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009dc0 tx=0x7fb4ac004b60 comp rx=0 tx=0).stop 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 msgr2=0x7fb4b002b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01215d0 0x7fb4b002b6a0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009dc0 tx=0x7fb4ac004b60 comp rx=0 tx=0).stop 2024-09-16T10:50:09.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4a6ffd700 10 client.4349.objecter ms_handle_connect 0x7fb494036f60 2024-09-16T10:50:09.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.908+0000 7fb4b6cfa700 1 --2- 172.21.15.5:0/1614159111 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb494036f60 0x7fb494039420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0123db0 tx=0x7fb4a8007d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:50:09.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 0x7fb4b002bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0067380 con 0x7fb4b01219b0 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 0x7fb4b002bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient: get_auth_request con 0x7fb4b01219b0 auth_method 0 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-16T10:50:09.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 1 -- 172.21.15.5:0/1614159111 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4b00a0070 con 0x7fb4b01219b0 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 client.4349.objecter ms_handle_connect 0x7fb4b01219b0 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 client.4349.objecter resend_mon_ops 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b5cf8700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 0x7fb4b002bbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009d90 tx=0x7fb4ac004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac02d880 con 0x7fb4b01219b0 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 monclient: dump: 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:09.933 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4614 (secure 0 0 0) 0x7fb4ac01b3f0 con 0x7fb4b01219b0 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 msgr2=0x7fb4b002bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 0x7fb4b002bbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009d90 tx=0x7fb4ac004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 msgr2=0x7fb4b002bbe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:09.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4a6ffd700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb4b01219b0 0x7fb4b002bbe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 12, 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 20, 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:09.935 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:09.936 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:50:09.938 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:09.939 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.943 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:50:09.944 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:09.945 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:09.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:09.947 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:50:09.947 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:09.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 librados: watch_flush enter 2024-09-16T10:50:09.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 librados: watch_flush exit 2024-09-16T10:50:09.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 20 client.4349.objecter shutdown clearing up homeless session... 2024-09-16T10:50:09.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 client.4349.objecter successfully canceled tick 2024-09-16T10:50:09.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb494036f60 msgr2=0x7fb494039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/1614159111 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb494036f60 0x7fb494039420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0123db0 tx=0x7fb4a8007d80 comp rx=0 tx=0).stop 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 10 monclient: shutdown 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 msgr2=0x7fb4b0137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000c530 tx=0x7fb4a000c840 comp rx=0 tx=0).stop 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b6cfa700 1 -- 172.21.15.5:0/1614159111 reap_dead start 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 shutdown_connections 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 --2- 172.21.15.5:0/1614159111 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb4b00a2c80 0x7fb4b0137d70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 >> 172.21.15.5:0/1614159111 conn(0x7fb4b00a4d80 msgr2=0x7fb4b00a6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 shutdown_connections 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 -- 172.21.15.5:0/1614159111 wait complete. 2024-09-16T10:50:09.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:09.920+0000 7fb4b74fb700 1 librados: shutdown 2024-09-16T10:50:09.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:50:09.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:50:09.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:50:10.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 10 monclient: get_monmap_and_config 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 10 monclient: monmap: 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:10.070563+0000 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:10.070563+0000 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:10.073 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:10.074 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:10.074 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 1 Processor -- start 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 1 -- start start 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.068+0000 7f931b59e700 10 monclient: init 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f931c09b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f931c09bd00 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f931c09c0e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c09fd30 con 0x7f931c09c0e0 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c1334d0 con 0x7f931c09bd00 2024-09-16T10:50:10.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c09c570 con 0x7f931c09b3e0 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient(hunting): authenticate will time out at 868.648568s 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931ad9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f931c09c0e0 0x7f931c132dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59126/0 (socket says 172.21.15.5:59126) 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38304/0 (socket says 172.21.15.5:38304) 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931ad9d700 1 -- 172.21.15.5:0/1172653406 learned_addr learned my addr 172.21.15.5:0/1172653406 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): get_auth_request con 0x7f931c09b3e0 auth_method 0 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830349361 0 0) 0x7f931c1334d0 con 0x7f931c09bd00 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient(hunting): _negotiate 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient(hunting): my global_id is 4417 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93080035d0 con 0x7f931c09bd00 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356720685 0 0) 0x7f931c09fd30 con 0x7f931c09c0e0 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 -- 172.21.15.5:0/1172653406 >> v1:172.21.15.143:6789/0 conn(0x7f931c09bd00 legacy=0x7f931c1326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 -- 172.21.15.5:0/1172653406 >> v1:172.21.15.5:6789/0 conn(0x7f931c09c0e0 legacy=0x7f931c132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 monclient: found mon.noname-b 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 -- 172.21.15.5:0/1172653406 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c09e350 con 0x7f931c09b3e0 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931a59c700 1 --2- 172.21.15.5:0/1172653406 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f930c009880 tx=0x7f930c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9330862a60b0ea4f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c011830 con 0x7f931c09b3e0 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: dump: 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.082 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 5 monclient: authenticate success, global_id 4352 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:10.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c011990 con 0x7f931c09b3e0 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 4 set_mon_vals no callback set 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 1 -- 172.21.15.5:0/1172653406 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c011d20 con 0x7f931c09b3e0 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931959a700 10 monclient: dump: 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.084 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 10 monclient: shutdown 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- 172.21.15.5:0/1172653406 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 msgr2=0x7f931c09b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 --2- 172.21.15.5:0/1172653406 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f930c009880 tx=0x7f930c009b90 comp rx=0 tx=0).stop 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- 172.21.15.5:0/1172653406 shutdown_connections 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 --2- 172.21.15.5:0/1172653406 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09b3e0 0x7f931c09b7c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- 172.21.15.5:0/1172653406 >> 172.21.15.5:0/1172653406 conn(0x7f931c0a45d0 msgr2=0x7f931c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- 172.21.15.5:0/1172653406 shutdown_connections 2024-09-16T10:50:10.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.072+0000 7f931b59e700 1 -- 172.21.15.5:0/1172653406 wait complete. 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient: monmap: 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:10.091 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 librados: starting msgr at 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 librados: starting objecter 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 Processor -- start 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 -- start start 2024-09-16T10:50:10.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 librados: setting wanted keys 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 librados: calling monclient init 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient: init 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 0x7f931c1c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f931c09b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09bd00 0x7f931c121030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f931c09bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f931c09c0e0 0x7f931c121570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f931c09c0e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0708f0 con 0x7f931c09b3e0 2024-09-16T10:50:10.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0a36d0 con 0x7f931c09c0e0 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c09b0d0 con 0x7f931c09bd00 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931b59e700 10 monclient(hunting): authenticate will time out at 868.655192s 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 0x7f931c1c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09bd00 0x7f931c121030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 0x7f931c1c8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49198/0 (socket says 172.21.15.5:49198) 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 1 -- 172.21.15.5:0/2475023162 learned_addr learned my addr 172.21.15.5:0/2475023162 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): get_auth_request con 0x7f931c09b3e0 auth_method 0 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): get_auth_request con 0x7f931c09bd00 auth_method 0 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f931c09c0e0 0x7f931c121570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): get_auth_request con 0x7f931c09c0e0 auth_method 0 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.076+0000 7f931ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09bd00 msgr2=0x7f931c121030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c09bd00 0x7f931c121030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f931c09c0e0 msgr2=0x7f931c121570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f931c09c0e0 0x7f931c121570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 monclient: found mon.noname-a 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0094b0 con 0x7f931c09b3e0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931ad9d700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f931c09c0e0 0x7f931c121570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931a59c700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 0x7f931c1c8a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f930c013040 tx=0x7f930c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 client.?.objecter ms_handle_connect 0x7f931c09b3e0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c019950 con 0x7f931c09b3e0 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: dump: 2024-09-16T10:50:10.097 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 5 monclient: authenticate success, global_id 4440 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c01e070 con 0x7f931c09b3e0 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 monclient: _renew_subs 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c123110 con 0x7f931c09b3e0 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c019cf0 con 0x7f931c09b3e0 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9318d99700 4 set_mon_vals no callback set 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: dump: 2024-09-16T10:50:10.099 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 monclient: _renew_subs 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c124cb0 con 0x7f931c09b3e0 2024-09-16T10:50:10.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 1 librados: init done 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 0x7f931c02b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c124e40 con 0x7f931c122280 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f930c025480 con 0x7f931c09b3e0 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 0x7f931c02b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient: get_auth_request con 0x7f931c122280 auth_method 0 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f931c09cfe0 con 0x7f931c122280 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 --2- 172.21.15.5:0/2475023162 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92fc036f60 0x7f92fc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f930c020070 con 0x7f931c09b3e0 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9319d9b700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 0x7f931c02b6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f931c1335d0 tx=0x7f930400fb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 client.4440.objecter ms_dispatch 0x7f931c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 3 client.4440.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 3 client.4440.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931ad9d700 1 --2- 172.21.15.5:0/2475023162 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92fc036f60 0x7f92fc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 20 client.4440.objecter dump_active .. 0 homeless 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 client.4440.objecter ms_handle_connect 0x7f931c122280 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 client.4440.objecter resend_mon_ops 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f931ad9d700 10 monclient: get_auth_request con 0x7f92fc036f60 auth_method 0 2024-09-16T10:50:10.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304005320 con 0x7f931c122280 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: dump: 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.103 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f9304005480 con 0x7f931c122280 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 msgr2=0x7f931c02b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 0x7f931c02b6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f931c1335d0 tx=0x7f930400fb00 comp rx=0 tx=0).stop 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 msgr2=0x7f931c02b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.080+0000 7f9312ffd700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c122280 0x7f931c02b6a0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f931c1335d0 tx=0x7f930400fb00 comp rx=0 tx=0).stop 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.084+0000 7f9312ffd700 10 client.4440.objecter ms_handle_connect 0x7f92fc036f60 2024-09-16T10:50:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.084+0000 7f931ad9d700 1 --2- 172.21.15.5:0/2475023162 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92fc036f60 0x7f92fc039420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9314001490 tx=0x7f9314001420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f931b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f931b59e700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 0x7f931c02bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f931b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c067380 con 0x7f931c1c8fd0 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 0x7f931c02bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient: get_auth_request con 0x7f931c1c8fd0 auth_method 0 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 1 -- 172.21.15.5:0/2475023162 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f931c11dd00 con 0x7f931c1c8fd0 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9319d9b700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 0x7f931c02bbe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f931c1330c0 tx=0x7f9304001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 10 client.4440.objecter ms_handle_connect 0x7f931c1c8fd0 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 10 client.4440.objecter resend_mon_ops 2024-09-16T10:50:10.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304022070 con 0x7f931c1c8fd0 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.092+0000 7f9312ffd700 10 monclient: dump: 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.108 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4446 (secure 0 0 0) 0x7f9304007650 con 0x7f931c1c8fd0 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 msgr2=0x7f931c02bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 0x7f931c02bbe0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f931c1330c0 tx=0x7f9304001150 comp rx=0 tx=0).stop 2024-09-16T10:50:10.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 msgr2=0x7f931c02bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f9312ffd700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f931c1c8fd0 0x7f931c02bbe0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.112 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:10.112 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:10.112 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 12, 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 20, 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:50:10.113 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:50:10.114 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:50:10.115 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:50:10.116 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.117 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.118 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:10.119 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.121 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.122 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:50:10.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:50:10.124 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:10.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 10 librados: watch_flush enter 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 10 librados: watch_flush exit 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 20 client.4440.objecter shutdown clearing up homeless session... 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 10 client.4440.objecter successfully canceled tick 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92fc036f60 msgr2=0x7f92fc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 --2- 172.21.15.5:0/2475023162 >> v2:172.21.15.143:6812/1098205034 conn(0x7f92fc036f60 0x7f92fc039420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9314001490 tx=0x7f9314001420 comp rx=0 tx=0).stop 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 10 monclient: shutdown 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 msgr2=0x7f931c1c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 --2- 172.21.15.5:0/2475023162 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f931c09b3e0 0x7f931c1c8a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f930c013040 tx=0x7f930c004770 comp rx=0 tx=0).stop 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931ad9d700 1 -- 172.21.15.5:0/2475023162 reap_dead start 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931ad9d700 1 -- 172.21.15.5:0/2475023162 reap_dead start 2024-09-16T10:50:10.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 shutdown_connections 2024-09-16T10:50:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 >> 172.21.15.5:0/2475023162 conn(0x7f931c0a45d0 msgr2=0x7f931c06f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 shutdown_connections 2024-09-16T10:50:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 -- 172.21.15.5:0/2475023162 wait complete. 2024-09-16T10:50:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.096+0000 7f931b59e700 1 librados: shutdown 2024-09-16T10:50:10.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:50:10.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:50:10.127 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:50:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: get_monmap_and_config 2024-09-16T10:50:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: monmap: 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:10.251183+0000 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:10.251183+0000 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:10.255 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 1 Processor -- start 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 1 -- start start 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: init 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 0x7faf8809f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-b con 0x7faf8809d6d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf8809dab0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf88138c60 0x7faf8813b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-a con 0x7faf88138c60 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.248+0000 7faf8e384700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf880a0050 con 0x7faf8809dab0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf8809e5a0 con 0x7faf88138c60 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8809fd70 con 0x7faf8809d6d0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient(hunting): authenticate will time out at 868.828046s 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf88138c60 0x7faf8813b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 0x7faf8809f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf7f7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faf8809dab0 0x7faf8809de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59134/0 (socket says 172.21.15.5:59134) 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf7f7fe700 1 -- 172.21.15.5:0/3426978664 learned_addr learned my addr 172.21.15.5:0/3426978664 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): get_auth_request con 0x7faf88138c60 auth_method 0 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): get_auth_request con 0x7faf8809d6d0 auth_method 0 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 -- 172.21.15.5:0/3426978664 >> v2:172.21.15.5:3300/0 conn(0x7faf88138c60 msgr2=0x7faf8813b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 --2- 172.21.15.5:0/3426978664 >> v2:172.21.15.5:3300/0 conn(0x7faf88138c60 0x7faf8813b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 -- 172.21.15.5:0/3426978664 >> v1:172.21.15.5:6789/0 conn(0x7faf8809dab0 legacy=0x7faf8809de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 monclient: found mon.noname-b 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 -- 172.21.15.5:0/3426978664 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf880a0330 con 0x7faf8809d6d0 2024-09-16T10:50:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf87fff700 1 --2- 172.21.15.5:0/3426978664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 0x7faf8809f3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf78009850 tx=0x7faf78009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8773ac8558ba43ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 1 -- 172.21.15.5:0/3426978664 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78011830 con 0x7faf8809d6d0 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: dump: 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.266 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 1 -- 172.21.15.5:0/3426978664 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf78011990 con 0x7faf8809d6d0 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 1 -- 172.21.15.5:0/3426978664 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78011d40 con 0x7faf8809d6d0 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 5 monclient: authenticate success, global_id 4364 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 4 set_mon_vals no callback set 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf877fe700 10 monclient: dump: 2024-09-16T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient: shutdown 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- 172.21.15.5:0/3426978664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 msgr2=0x7faf8809f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 --2- 172.21.15.5:0/3426978664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 0x7faf8809f3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf78009850 tx=0x7faf78009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- 172.21.15.5:0/3426978664 shutdown_connections 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 --2- 172.21.15.5:0/3426978664 >> v2:172.21.15.5:3300/0 conn(0x7faf88138c60 0x7faf8813b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 --2- 172.21.15.5:0/3426978664 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809d6d0 0x7faf8809f3f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- 172.21.15.5:0/3426978664 >> 172.21.15.5:0/3426978664 conn(0x7faf880a0b30 msgr2=0x7faf880a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- 172.21.15.5:0/3426978664 shutdown_connections 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 -- 172.21.15.5:0/3426978664 wait complete. 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 10 monclient: monmap: 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 librados: starting msgr at 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.252+0000 7faf8e384700 1 librados: starting objecter 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 Processor -- start 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- start start 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 librados: setting wanted keys 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 librados: calling monclient init 2024-09-16T10:50:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: init 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8809d6d0 0x7faf88133420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-b con 0x7faf8809d6d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809dab0 0x7faf88133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-c con 0x7faf8809dab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): picked mon.noname-a con 0x7faf88138c60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8811e100 con 0x7faf88138c60 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8811e370 con 0x7faf8809d6d0 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8813ae70 con 0x7faf8809dab0 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient(hunting): authenticate will time out at 868.832584s 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809dab0 0x7faf88133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49222/0 (socket says 172.21.15.5:49222) 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 -- 172.21.15.5:0/2792606097 learned_addr learned my addr 172.21.15.5:0/2792606097 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): get_auth_request con 0x7faf88138c60 auth_method 0 2024-09-16T10:50:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): get_auth_request con 0x7faf8809dab0 auth_method 0 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8809d6d0 0x7faf88133420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): get_auth_request con 0x7faf8809d6d0 auth_method 0 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809dab0 msgr2=0x7faf88133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809dab0 0x7faf88133960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8809d6d0 msgr2=0x7faf88133420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8809d6d0 0x7faf88133420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 monclient: found mon.noname-a 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf780094b0 con 0x7faf88138c60 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf7f7fe700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf8809dab0 0x7faf88133960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8c921700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faf80009dd0 tx=0x7faf8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 client.?.objecter ms_handle_connect 0x7faf88138c60 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80009c40 con 0x7faf88138c60 2024-09-16T10:50:10.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: dump: 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 5 monclient: authenticate success, global_id 4449 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _renew_subs 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8000f040 con 0x7faf88138c60 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf88139320 con 0x7faf88138c60 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8000ceb0 con 0x7faf88138c60 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:10.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf86ffd700 4 set_mon_vals no callback set 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf857fa700 10 monclient: dump: 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _renew_subs 2024-09-16T10:50:10.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8811bfa0 con 0x7faf88138c60 2024-09-16T10:50:10.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 librados: init done 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.256+0000 7faf8e384700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 0x7faf8811afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8811e500 con 0x7faf8811abc0 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7faf80014480 con 0x7faf88138c60 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 0x7faf8811afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient: get_auth_request con 0x7faf8811abc0 auth_method 0 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 --2- 172.21.15.5:0/2792606097 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf68036f90 0x7faf68039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7faf800481f0 con 0x7faf88138c60 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 client.4449.objecter ms_dispatch 0x7faf88000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 3 client.4449.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 3 client.4449.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 20 client.4449.objecter dump_active .. 0 homeless 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf7f7fe700 1 --2- 172.21.15.5:0/2792606097 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf68036f90 0x7faf68039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf7f7fe700 10 monclient: get_auth_request con 0x7faf68036f90 auth_method 0 2024-09-16T10:50:10.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faf8813b590 con 0x7faf8811abc0 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf87fff700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 0x7faf8811afa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf78009850 tx=0x7faf78004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 client.4449.objecter ms_handle_connect 0x7faf8811abc0 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 client.4449.objecter resend_mon_ops 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 client.4449.objecter ms_handle_connect 0x7faf68036f90 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf7f7fe700 1 --2- 172.21.15.5:0/2792606097 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf68036f90 0x7faf68039450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf880706e0 tx=0x7faf70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78020070 con 0x7faf8811abc0 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 monclient: dump: 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.284 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7faf78004330 con 0x7faf8811abc0 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 msgr2=0x7faf8811afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 0x7faf8811afa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf78009850 tx=0x7faf78004770 comp rx=0 tx=0).stop 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 msgr2=0x7faf8811afa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.260+0000 7faf857fa700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811abc0 0x7faf8811afa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 0x7faf8806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88067580 con 0x7faf8811b4e0 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 0x7faf8806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient: get_auth_request con 0x7faf8811b4e0 auth_method 0 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 1 -- 172.21.15.5:0/2792606097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faf8809f3f0 con 0x7faf8811b4e0 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf87fff700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 0x7faf8806c6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf78000c00 tx=0x7faf78011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 client.4449.objecter ms_handle_connect 0x7faf8811b4e0 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 client.4449.objecter resend_mon_ops 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7802c830 con 0x7faf8811b4e0 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 monclient: dump: 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4778 (secure 0 0 0) 0x7faf78030070 con 0x7faf8811b4e0 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 msgr2=0x7faf8806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 0x7faf8806c6a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf78000c00 tx=0x7faf78011000 comp rx=0 tx=0).stop 2024-09-16T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 msgr2=0x7faf8806c6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf857fa700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf8811b4e0 0x7faf8806c6a0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf78000c00 tx=0x7faf78011000 comp rx=0 tx=0).stop 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 12, 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:50:10.289 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 20, 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:10.292 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:10.293 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:10.294 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.295 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.296 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.297 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.298 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:10.299 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:50:10.300 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:50:10.301 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:50:10.302 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 librados: watch_flush enter 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 librados: watch_flush exit 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 20 client.4449.objecter shutdown clearing up homeless session... 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 client.4449.objecter successfully canceled tick 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf68036f90 msgr2=0x7faf68039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 --2- 172.21.15.5:0/2792606097 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf68036f90 0x7faf68039450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf880706e0 tx=0x7faf70009040 comp rx=0 tx=0).stop 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 10 monclient: shutdown 2024-09-16T10:50:10.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8c921700 1 -- 172.21.15.5:0/2792606097 reap_dead start 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 msgr2=0x7faf8811a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faf80009dd0 tx=0x7faf8000c3a0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 shutdown_connections 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 --2- 172.21.15.5:0/2792606097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf88138c60 0x7faf8811a680 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 >> 172.21.15.5:0/2792606097 conn(0x7faf880a0b30 msgr2=0x7faf880a2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 shutdown_connections 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 -- 172.21.15.5:0/2792606097 wait complete. 2024-09-16T10:50:10.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.272+0000 7faf8e384700 1 librados: shutdown 2024-09-16T10:50:10.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:50:10.306 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T10:50:10.306 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:50:10.306 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T10:50:10.306 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T10:50:10.306 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T10:50:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: get_monmap_and_config 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: monmap: 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:10.426483+0000 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:10.426483+0000 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:10.430 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 1 Processor -- start 2024-09-16T10:50:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 1 -- start start 2024-09-16T10:50:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: init 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ce809b960 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2ce80a2700 0x7f2ce813b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-a con 0x7f2ce80a2700 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.424+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-b con 0x7f2ce813b780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80a4c90 con 0x7f2ce809b960 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80a6120 con 0x7f2ce80a2700 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80a1740 con 0x7f2ce813b780 2024-09-16T10:50:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient(hunting): authenticate will time out at 869.004090s 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2ce80a2700 0x7f2ce813b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced9ee700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2ce809b960 0x7f2ce809bd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:51566/0 (socket says 172.21.15.5:51566) 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced9ee700 1 -- 172.21.15.5:0/3321881667 learned_addr learned my addr 172.21.15.5:0/3321881667 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38330/0 (socket says 172.21.15.5:38330) 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): get_auth_request con 0x7f2ce813b780 auth_method 0 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): get_auth_request con 0x7f2ce80a2700 auth_method 0 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 -- 172.21.15.5:0/3321881667 >> v2:172.21.15.5:3300/0 conn(0x7f2ce80a2700 msgr2=0x7f2ce813b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 --2- 172.21.15.5:0/3321881667 >> v2:172.21.15.5:3300/0 conn(0x7f2ce80a2700 0x7f2ce813b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 -- 172.21.15.5:0/3321881667 >> v1:172.21.15.143:6789/0 conn(0x7f2ce809b960 legacy=0x7f2ce809bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 monclient: found mon.noname-b 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 -- 172.21.15.5:0/3321881667 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80a1d70 con 0x7f2ce813b780 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cee1ef700 1 --2- 172.21.15.5:0/3321881667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009d90 tx=0x7f2ce000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40bc8be4b217a68d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 1 -- 172.21.15.5:0/3321881667 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000e070 con 0x7f2ce813b780 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: dump: 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 5 monclient: authenticate success, global_id 4370 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 1 -- 172.21.15.5:0/3321881667 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce000bb20 con 0x7f2ce813b780 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 4 set_mon_vals no callback set 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 1 -- 172.21.15.5:0/3321881667 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0012430 con 0x7f2ce813b780 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cec9ec700 10 monclient: dump: 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient: shutdown 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- 172.21.15.5:0/3321881667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 msgr2=0x7f2ce813db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 --2- 172.21.15.5:0/3321881667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009d90 tx=0x7f2ce000b330 comp rx=0 tx=0).stop 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- 172.21.15.5:0/3321881667 shutdown_connections 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 --2- 172.21.15.5:0/3321881667 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce813b780 0x7f2ce813db70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 --2- 172.21.15.5:0/3321881667 >> v2:172.21.15.5:3300/0 conn(0x7f2ce80a2700 0x7f2ce813b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- 172.21.15.5:0/3321881667 >> 172.21.15.5:0/3321881667 conn(0x7f2ce80a5330 msgr2=0x7f2ce80a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- 172.21.15.5:0/3321881667 shutdown_connections 2024-09-16T10:50:10.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 -- 172.21.15.5:0/3321881667 wait complete. 2024-09-16T10:50:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 10 monclient: monmap: 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 librados: starting msgr at 2024-09-16T10:50:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.428+0000 7f2cefc52700 1 librados: starting objecter 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 Processor -- start 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- start start 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 librados: setting wanted keys 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 librados: calling monclient init 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: init 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce809b960 0x7f2ce8098aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-b con 0x7f2ce809b960 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce80a2700 0x7f2ce806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-c con 0x7f2ce80a2700 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 0x7f2ce806e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): picked mon.noname-a con 0x7f2ce813b780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80998a0 con 0x7f2ce813b780 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce806fbf0 con 0x7f2ce809b960 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce809fa20 con 0x7f2ce80a2700 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient(hunting): authenticate will time out at 869.009082s 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce80a2700 0x7f2ce806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 0x7f2ce806e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 0x7f2ce806e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49238/0 (socket says 172.21.15.5:49238) 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce80a2700 0x7f2ce806dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38346/0 (socket says 172.21.15.5:38346) 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 learned_addr learned my addr 172.21.15.5:0/4290826575 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): get_auth_request con 0x7f2ce813b780 auth_method 0 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): get_auth_request con 0x7f2ce80a2700 auth_method 0 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce809b960 0x7f2ce8098aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): get_auth_request con 0x7f2ce809b960 auth_method 0 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced1ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce80a2700 msgr2=0x7f2ce806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2ce80a2700 0x7f2ce806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce809b960 msgr2=0x7f2ce8098aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce809b960 0x7f2ce8098aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 monclient: found mon.noname-a 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce0009a70 con 0x7f2ce813b780 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cee1ef700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 0x7f2ce806e120 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0014040 tx=0x7f2ce0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 client.?.objecter ms_handle_connect 0x7f2ce813b780 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0012430 con 0x7f2ce813b780 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: dump: 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0020070 con 0x7f2ce813b780 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 5 monclient: authenticate success, global_id 4455 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _renew_subs 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce8071760 con 0x7f2ce813b780 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0004280 con 0x7f2ce813b780 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cde7fc700 10 monclient: dump: 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.456 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cdffff700 4 set_mon_vals no callback set 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _renew_subs 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce8071ab0 con 0x7f2ce813b780 2024-09-16T10:50:10.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.432+0000 7f2cefc52700 1 librados: init done 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2ce00043e0 con 0x7f2ce813b780 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cefc52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cefc52700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 0x7f2ce806e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8071260 con 0x7f2ce8098fe0 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 --2- 172.21.15.5:0/4290826575 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2ccc036f60 0x7f2ccc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f2ce0022070 con 0x7f2ce813b780 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 client.4455.objecter ms_dispatch 0x7f2ce8000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 3 client.4455.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 3 client.4455.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 20 client.4455.objecter dump_active .. 0 homeless 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced1ed700 1 --2- 172.21.15.5:0/4290826575 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2ccc036f60 0x7f2ccc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced1ed700 10 monclient: get_auth_request con 0x7f2ccc036f60 auth_method 0 2024-09-16T10:50:10.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 0x7f2ce806e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient: get_auth_request con 0x7f2ce8098fe0 auth_method 0 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 client.4455.objecter ms_handle_connect 0x7f2ccc036f60 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced1ed700 1 --2- 172.21.15.5:0/4290826575 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2ccc036f60 0x7f2ccc039420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ce806f360 tx=0x7f2cd8009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ce80a14f0 con 0x7f2ce8098fe0 2024-09-16T10:50:10.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 client.4455.objecter ms_handle_connect 0x7f2ce8098fe0 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 client.4455.objecter resend_mon_ops 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2ced9ee700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 0x7f2ce806e660 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4000f60 tx=0x7f2ce400fb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4005b70 con 0x7f2ce8098fe0 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 monclient: dump: 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.460 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2ce4005cd0 con 0x7f2ce8098fe0 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 msgr2=0x7f2ce806e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 0x7f2ce806e660 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4000f60 tx=0x7f2ce400fb00 comp rx=0 tx=0).stop 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 msgr2=0x7f2ce806e660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.436+0000 7f2cde7fc700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce8098fe0 0x7f2ce806e660 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4000f60 tx=0x7f2ce400fb00 comp rx=0 tx=0).stop 2024-09-16T10:50:10.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2cefc52700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T10:50:10.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2cefc52700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 0x7f2ce809c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2cefc52700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80677c0 con 0x7f2ce806eba0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 0x7f2ce809c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient: get_auth_request con 0x7f2ce806eba0 auth_method 0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.448+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2ced9ee700 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2ced9ee700 1 -- 172.21.15.5:0/4290826575 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f2ce8061fa0 con 0x7f2ce806eba0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2ced9ee700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 0x7f2ce809c660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400fb40 tx=0x7f2ce40058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 client.4455.objecter ms_handle_connect 0x7f2ce806eba0 2024-09-16T10:50:10.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 client.4455.objecter resend_mon_ops 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4022070 con 0x7f2ce806eba0 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 monclient: dump: 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.465 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (secure 0 0 0) 0x7f2ce4005e40 con 0x7f2ce806eba0 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 msgr2=0x7f2ce809c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 0x7f2ce809c660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400fb40 tx=0x7f2ce40058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 msgr2=0x7f2ce809c660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cde7fc700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce806eba0 0x7f2ce809c660 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400fb40 tx=0x7f2ce40058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:10.466 INFO:teuthology.orchestra.run.smithi005.stdout: "ret": 0, 2024-09-16T10:50:10.467 INFO:teuthology.orchestra.run.smithi005.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T10:50:10.467 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 10 librados: watch_flush enter 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 10 librados: watch_flush exit 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 20 client.4455.objecter shutdown clearing up homeless session... 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 10 client.4455.objecter successfully canceled tick 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2ccc036f60 msgr2=0x7f2ccc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 --2- 172.21.15.5:0/4290826575 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2ccc036f60 0x7f2ccc039420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ce806f360 tx=0x7f2cd8009240 comp rx=0 tx=0).stop 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 10 monclient: shutdown 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 msgr2=0x7f2ce806e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 --2- 172.21.15.5:0/4290826575 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ce813b780 0x7f2ce806e120 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0014040 tx=0x7f2ce0004af0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 reap_dead start 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cee1ef700 1 -- 172.21.15.5:0/4290826575 reap_dead start 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 shutdown_connections 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 >> 172.21.15.5:0/4290826575 conn(0x7f2ce80a5330 msgr2=0x7f2ce80a6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 shutdown_connections 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 -- 172.21.15.5:0/4290826575 wait complete. 2024-09-16T10:50:10.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.452+0000 7f2cefc52700 1 librados: shutdown 2024-09-16T10:50:10.476 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:50:10.476 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T10:50:10.476 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:50:10.515 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T10:50:10.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:50:10.516 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.628+0000 7f20d0c8c700 10 monclient: get_monmap_and_config 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.628+0000 7f20d0c8c700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.628+0000 7f20d0c8c700 10 monclient: monmap: 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:10.632578+0000 2024-09-16T10:50:10.635 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:10.632578+0000 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 Processor -- start 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- start start 2024-09-16T10:50:10.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: init 2024-09-16T10:50:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T10:50:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20cc09e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:10.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20cc09d6a0 0x7f20cc138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-a con 0x7f20cc09d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 0x7f20cc13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-c con 0x7f20cc09dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc070e40 con 0x7f20cc09e8e0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0a01d0 con 0x7f20cc09d6a0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc09fef0 con 0x7f20cc09dfd0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient(hunting): authenticate will time out at 869.208937s 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20cc09d6a0 0x7f20cc138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f20cc09d6a0 0x7f20cc138b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49246/0 (socket says 172.21.15.5:49246) 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 -- 172.21.15.5:0/2326308153 learned_addr learned my addr 172.21.15.5:0/2326308153 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): get_auth_request con 0x7f20cc09d6a0 auth_method 0 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20cb7fe700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f20cc09e8e0 legacy=0x7f20cc09ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20cbfff700 1 -- 172.21.15.5:0/2326308153 >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 msgr2=0x7f20cc13b0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20cbfff700 1 --2- 172.21.15.5:0/2326308153 >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 0x7f20cc13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 -- 172.21.15.5:0/2326308153 >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 msgr2=0x7f20cc13b0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 --2- 172.21.15.5:0/2326308153 >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 0x7f20cc13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 -- 172.21.15.5:0/2326308153 >> v1:172.21.15.143:6789/0 conn(0x7f20cc09e8e0 legacy=0x7f20cc09ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 monclient: found mon.noname-a 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 -- 172.21.15.5:0/2326308153 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc09f9f0 con 0x7f20cc09d6a0 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20caffd700 1 --2- 172.21.15.5:0/2326308153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09d6a0 0x7f20cc138b80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009910 tx=0x7f20b4009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d807f36483dc36a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 1 -- 172.21.15.5:0/2326308153 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4004230 con 0x7f20cc09d6a0 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:10.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: dump: 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 5 monclient: authenticate success, global_id 4458 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 1 -- 172.21.15.5:0/2326308153 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b4004390 con 0x7f20cc09d6a0 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 4 set_mon_vals no callback set 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 1 -- 172.21.15.5:0/2326308153 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4009e60 con 0x7f20cc09d6a0 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20ca7fc700 10 monclient: dump: 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.647 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: shutdown 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- 172.21.15.5:0/2326308153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09d6a0 msgr2=0x7f20cc138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/2326308153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09d6a0 0x7f20cc138b80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009910 tx=0x7f20b4009c20 comp rx=0 tx=0).stop 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- 172.21.15.5:0/2326308153 shutdown_connections 2024-09-16T10:50:10.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/2326308153 >> v2:172.21.15.143:3300/0 conn(0x7f20cc09dfd0 0x7f20cc13b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/2326308153 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09d6a0 0x7f20cc138b80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- 172.21.15.5:0/2326308153 >> 172.21.15.5:0/2326308153 conn(0x7f20cc0a45d0 msgr2=0x7f20cc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- 172.21.15.5:0/2326308153 shutdown_connections 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 -- 172.21.15.5:0/2326308153 wait complete. 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: build_initial_monmap 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 10 monclient: monmap: 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:10.649 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 librados: starting msgr at 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.632+0000 7f20d0c8c700 1 librados: starting objecter 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 Processor -- start 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- start start 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 librados: setting wanted keys 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 librados: calling monclient init 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: init 2024-09-16T10:50:10.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 0x7f20cc1c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-b con 0x7f20cc09d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 0x7f20cc133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-c con 0x7f20cc09dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 0x7f20cc133fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): picked mon.noname-a con 0x7f20cc09e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc09b0d0 con 0x7f20cc09e8e0 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc11fd40 con 0x7f20cc09d6a0 2024-09-16T10:50:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc06ce70 con 0x7f20cc09dfd0 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient(hunting): authenticate will time out at 869.212703s 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cb7fe700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 msgr2=0x7f20cc1c8620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 0x7f20cc133fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cb7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 0x7f20cc1c8620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 0x7f20cc133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 0x7f20cc133fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:49254/0 (socket says 172.21.15.5:49254) 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): get_auth_request con 0x7f20cc09dfd0 auth_method 0 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 1 -- 172.21.15.5:0/4115533079 learned_addr learned my addr 172.21.15.5:0/4115533079 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): get_auth_request con 0x7f20cc09e8e0 auth_method 0 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-16T10:50:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 -- 172.21.15.5:0/4115533079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 msgr2=0x7f20cc1c8620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 0x7f20cc1c8620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 -- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 msgr2=0x7f20cc133fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 0x7f20cc133fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 monclient: found mon.noname-c 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 -- 172.21.15.5:0/4115533079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4009570 con 0x7f20cc09dfd0 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20caffd700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 0x7f20cc133aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20b4015000 tx=0x7f20b4004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 client.?.objecter ms_handle_connect 0x7f20cc09dfd0 2024-09-16T10:50:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b40045c0 con 0x7f20cc09dfd0 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: dump: 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.656 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 5 monclient: authenticate success, global_id 4376 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _renew_subs 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b401f070 con 0x7f20cc09dfd0 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc121180 con 0x7f20cc09dfd0 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4003900 con 0x7f20cc09dfd0 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:10.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 10 monclient: dump: 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c97fa700 4 set_mon_vals no callback set 2024-09-16T10:50:10.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _renew_subs 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc11f100 con 0x7f20cc09dfd0 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 librados: init done 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc09ecc0 con 0x7f20cc09dfd0 2024-09-16T10:50:10.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.636+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f20b4021030 con 0x7f20cc09dfd0 2024-09-16T10:50:10.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 1 --2- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 0x7f20ac0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:10.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f20b4048e00 con 0x7f20cc09dfd0 2024-09-16T10:50:10.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 10 client.4376.objecter ms_dispatch 0x7f20cc000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 3 client.4376.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 3 client.4376.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 20 client.4376.objecter dump_active .. 0 homeless 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20cb7fe700 1 --2- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 0x7f20ac0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20cb7fe700 10 monclient: get_auth_request con 0x7f20ac036f10 auth_method 0 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20cb7fe700 1 --2- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 0x7f20ac0393d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0039e0 tx=0x7f20bc009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.640+0000 7f20c3fff700 10 client.4376.objecter ms_handle_connect 0x7f20ac036f10 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.644+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f20b4019d90 con 0x7f20cc09dfd0 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.644+0000 7f20c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:10.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.644+0000 7f20c3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20d0c8c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20d0c8c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20d0c8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f20cc09ecc0 con 0x7f20cc09dfd0 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20c3fff700 1 -- 172.21.15.5:0/4115533079 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7f20b4019bb0 con 0x7f20cc09dfd0 2024-09-16T10:50:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.884+0000 7f20c3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:10.890 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 10 librados: watch_flush enter 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 10 librados: watch_flush exit 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 20 client.4376.objecter shutdown clearing up homeless session... 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 10 client.4376.objecter successfully canceled tick 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 msgr2=0x7f20ac0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 0x7f20ac0393d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0039e0 tx=0x7f20bc009270 comp rx=0 tx=0).stop 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 10 monclient: shutdown 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 msgr2=0x7f20cc133aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 0x7f20cc133aa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20b4015000 tx=0x7f20b4004770 comp rx=0 tx=0).stop 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 shutdown_connections 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> v2:172.21.15.143:6812/1098205034 conn(0x7f20ac036f10 0x7f20ac0393d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f20cc09e8e0 0x7f20cc133fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20cc09dfd0 0x7f20cc133aa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 --2- 172.21.15.5:0/4115533079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20cc09d6a0 0x7f20cc1c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 >> 172.21.15.5:0/4115533079 conn(0x7f20cc0a45d0 msgr2=0x7f20cc06fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 shutdown_connections 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 -- 172.21.15.5:0/4115533079 wait complete. 2024-09-16T10:50:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:10.888+0000 7f20d0c8c700 1 librados: shutdown 2024-09-16T10:50:13.906 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:50:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: get_monmap_and_config 2024-09-16T10:50:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: build_initial_monmap 2024-09-16T10:50:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: monmap: 2024-09-16T10:50:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:14.023106+0000 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:14.023106+0000 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:14.027 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:14.027 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 Processor -- start 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 -- start start 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: init 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f453409b3e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-b con 0x7f453409bed0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 0x7f4534139e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-c con 0x7f453409c2b0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453409fd30 con 0x7f453409b3e0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f453409e350 con 0x7f453409c2b0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453409ca20 con 0x7f453409bed0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453ac43700 10 monclient(hunting): authenticate will time out at 872.599481s 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539c41700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f453409b3e0 legacy=0x7f453409b7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453a442700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 msgr2=0x7f4534139e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60866/0 (socket says 172.21.15.5:60866) 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 1 -- 172.21.15.5:0/307717091 learned_addr learned my addr 172.21.15.5:0/307717091 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f453a442700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 0x7f4534139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): get_auth_request con 0x7f453409bed0 auth_method 0 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:14.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.020+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 -- 172.21.15.5:0/307717091 >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 msgr2=0x7f4534139e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 --2- 172.21.15.5:0/307717091 >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 0x7f4534139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 -- 172.21.15.5:0/307717091 >> v1:172.21.15.143:6789/0 conn(0x7f453409b3e0 legacy=0x7f453409b7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 10 monclient: _finish_hunting 0 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 monclient: found mon.noname-b 2024-09-16T10:50:14.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 -- 172.21.15.5:0/307717091 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453409cd00 con 0x7f453409bed0 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4539440700 1 --2- 172.21.15.5:0/307717091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4530009910 tx=0x7f4530009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31bed3bb8782dd06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 1 -- 172.21.15.5:0/307717091 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530004230 con 0x7f453409bed0 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: dump: 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.040 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: _finish_auth 0 2024-09-16T10:50:14.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: _check_auth_tickets 2024-09-16T10:50:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 1 -- 172.21.15.5:0/307717091 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4530004390 con 0x7f453409bed0 2024-09-16T10:50:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 1 -- 172.21.15.5:0/307717091 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530009e60 con 0x7f453409bed0 2024-09-16T10:50:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 5 monclient: authenticate success, global_id 4379 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 4 set_mon_vals no callback set 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f4538c3f700 10 monclient: dump: 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: shutdown 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:14.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- 172.21.15.5:0/307717091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 msgr2=0x7f45341378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- 172.21.15.5:0/307717091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4530009910 tx=0x7f4530009c20 comp rx=0 tx=0).stop 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- 172.21.15.5:0/307717091 shutdown_connections 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- 172.21.15.5:0/307717091 >> v2:172.21.15.143:3300/0 conn(0x7f453409c2b0 0x7f4534139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- 172.21.15.5:0/307717091 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341378f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- 172.21.15.5:0/307717091 >> 172.21.15.5:0/307717091 conn(0x7f45340a45d0 msgr2=0x7f45340a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- 172.21.15.5:0/307717091 shutdown_connections 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- 172.21.15.5:0/307717091 wait complete. 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: build_initial_monmap 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: monmap: 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:14.045 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 librados: starting msgr at 2024-09-16T10:50:14.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 librados: starting objecter 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 Processor -- start 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- start start 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 librados: setting wanted keys 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 librados: calling monclient init 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: init 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 0x7f45341cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-b con 0x7f453409b3e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-c con 0x7f453409bed0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): picked mon.noname-a con 0x7f453409c2b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45341395e0 con 0x7f453409c2b0 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453409b0d0 con 0x7f453409b3e0 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45341282b0 con 0x7f453409bed0 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.024+0000 7f453ac43700 10 monclient(hunting): authenticate will time out at 872.603934s 2024-09-16T10:50:14.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539c41700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 msgr2=0x7f45341cf5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59222/0 (socket says 172.21.15.5:59222) 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 -- 172.21.15.5:0/2531373665 learned_addr learned my addr 172.21.15.5:0/2531373665 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): get_auth_request con 0x7f453409c2b0 auth_method 0 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539c41700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 0x7f45341cf5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): get_auth_request con 0x7f453409bed0 auth_method 0 2024-09-16T10:50:14.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f4539440700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-16T10:50:14.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 -- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 msgr2=0x7f45341cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 -- 172.21.15.5:0/2531373665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 msgr2=0x7f45341cf5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 0x7f45341cf5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient: _finish_hunting 0 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 monclient: found mon.noname-a 2024-09-16T10:50:14.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 -- 172.21.15.5:0/2531373665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530009570 con 0x7f453409c2b0 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 client.?.objecter ms_handle_connect 0x7f453409c2b0 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453a442700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f452c00b990 tx=0x7f452c00bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c00d8e0 con 0x7f453409c2b0 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:14.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: dump: 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:14.053 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 5 monclient: authenticate success, global_id 4464 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452c006e90 con 0x7f453409c2b0 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _renew_subs 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45341285f0 con 0x7f453409c2b0 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c00dc60 con 0x7f453409c2b0 2024-09-16T10:50:14.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 monclient: dump: 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:14.055 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452bfff700 4 set_mon_vals no callback set 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _renew_subs 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4534126230 con 0x7f453409c2b0 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 1 librados: init done 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45340a1dc0 con 0x7f453409c2b0 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f452c00c400 con 0x7f453409c2b0 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 --2- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 0x7f45180393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f452c047650 con 0x7f453409c2b0 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 10 client.4464.objecter ms_dispatch 0x7f4534000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 3 client.4464.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 3 client.4464.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.028+0000 7f452a7fc700 20 client.4464.objecter dump_active .. 0 homeless 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.032+0000 7f4539c41700 1 --2- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 0x7f45180393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.032+0000 7f4539c41700 10 monclient: get_auth_request con 0x7f4518036f10 auth_method 0 2024-09-16T10:50:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.032+0000 7f4539c41700 1 --2- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 0x7f45180393d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f45240039e0 tx=0x7f4524009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.032+0000 7f452a7fc700 10 client.4464.objecter ms_handle_connect 0x7f4518036f10 2024-09-16T10:50:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.036+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f452c014360 con 0x7f453409c2b0 2024-09-16T10:50:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.036+0000 7f452a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:14.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.036+0000 7f452a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f453ac43700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f453ac43700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f453ac43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f453409b7c0 con 0x7f453409c2b0 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f452a7fc700 1 -- 172.21.15.5:0/2531373665 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7f452c01a070 con 0x7f453409c2b0 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f452a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.268+0000 7f452a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:14.274 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:50:14.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 10 librados: watch_flush enter 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 10 librados: watch_flush exit 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 20 client.4464.objecter shutdown clearing up homeless session... 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 10 client.4464.objecter successfully canceled tick 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 msgr2=0x7f45180393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 0x7f45180393d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f45240039e0 tx=0x7f4524009270 comp rx=0 tx=0).stop 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 10 monclient: shutdown 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 msgr2=0x7f4534124590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f452c00b990 tx=0x7f452c00bca0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 shutdown_connections 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4518036f10 0x7f45180393d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f453409c2b0 0x7f4534124590 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f453409bed0 0x7f45341cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 --2- 172.21.15.5:0/2531373665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f453409b3e0 0x7f45341cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 >> 172.21.15.5:0/2531373665 conn(0x7f45340a45d0 msgr2=0x7f453409e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 shutdown_connections 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 -- 172.21.15.5:0/2531373665 wait complete. 2024-09-16T10:50:14.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:14.272+0000 7f453ac43700 1 librados: shutdown 2024-09-16T10:50:17.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:50:17.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 10 monclient: get_monmap_and_config 2024-09-16T10:50:17.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 10 monclient: build_initial_monmap 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 10 monclient: monmap: 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:17.407471+0000 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:17.407471+0000 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:17.410 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:17.411 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 1 Processor -- start 2024-09-16T10:50:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 1 -- start start 2024-09-16T10:50:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.404+0000 7fa616bcd700 10 monclient: init 2024-09-16T10:50:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 0x7fa608021060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-c con 0x7fa608006720 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa608006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-b con 0x7fa608006b00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6080b8490 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080215a0 con 0x7fa6080b8490 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60801ee40 con 0x7fa608006720 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60801f270 con 0x7fa608006b00 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:17.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient(hunting): authenticate will time out at 875.984634s 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6163cc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa6080b8490 0x7fa6080b8920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57946/0 (socket says 172.21.15.5:57946) 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6163cc700 1 -- 172.21.15.5:0/33053051 learned_addr learned my addr 172.21.15.5:0/33053051 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 --2- 172.21.15.5:0/33053051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa608006f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa615bcb700 1 -- 172.21.15.5:0/33053051 >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 msgr2=0x7fa608021060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa615bcb700 1 --2- 172.21.15.5:0/33053051 >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 0x7fa608021060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): get_auth_request con 0x7fa608006b00 auth_method 0 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:17.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 -- 172.21.15.5:0/33053051 >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 msgr2=0x7fa608021060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 --2- 172.21.15.5:0/33053051 >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 0x7fa608021060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 -- 172.21.15.5:0/33053051 >> v1:172.21.15.5:6789/0 conn(0x7fa6080b8490 legacy=0x7fa6080b8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient: _finish_hunting 0 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 monclient: found mon.noname-b 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 -- 172.21.15.5:0/33053051 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60801fde0 con 0x7fa608006b00 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa6153ca700 1 --2- 172.21.15.5:0/33053051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa608006f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa600009850 tx=0x7fa600009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ac021f48d120ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 1 -- 172.21.15.5:0/33053051 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600011830 con 0x7fa608006b00 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: dump: 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: _finish_auth 0 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: _check_auth_tickets 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 5 monclient: authenticate success, global_id 4385 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 1 -- 172.21.15.5:0/33053051 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa600011990 con 0x7fa608006b00 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 4 set_mon_vals no callback set 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 1 -- 172.21.15.5:0/33053051 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600011d20 con 0x7fa608006b00 2024-09-16T10:50:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa614bc9700 10 monclient: dump: 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: shutdown 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- 172.21.15.5:0/33053051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 msgr2=0x7fa608006f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 --2- 172.21.15.5:0/33053051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa608006f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa600009850 tx=0x7fa600009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- 172.21.15.5:0/33053051 shutdown_connections 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 --2- 172.21.15.5:0/33053051 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa608006f20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 --2- 172.21.15.5:0/33053051 >> v2:172.21.15.143:3300/0 conn(0x7fa608006720 0x7fa608021060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- 172.21.15.5:0/33053051 >> 172.21.15.5:0/33053051 conn(0x7fa608027320 msgr2=0x7fa608027700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- 172.21.15.5:0/33053051 shutdown_connections 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 1 -- 172.21.15.5:0/33053051 wait complete. 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: build_initial_monmap 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.408+0000 7fa616bcd700 10 monclient: monmap: 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:17.422 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 librados: starting msgr at 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 librados: starting objecter 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 Processor -- start 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- start start 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 librados: setting wanted keys 2024-09-16T10:50:17.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 librados: calling monclient init 2024-09-16T10:50:17.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: init 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-a con 0x7fa608006720 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa6080b7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-c con 0x7fa608006b00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 0x7fa6080b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): picked mon.noname-b con 0x7fa6080b8490 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608150950 con 0x7fa608006720 2024-09-16T10:50:17.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60802a430 con 0x7fa6080b8490 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60801e030 con 0x7fa608006b00 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient(hunting): authenticate will time out at 875.989224s 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59234/0 (socket says 172.21.15.5:59234) 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 -- 172.21.15.5:0/3210012480 learned_addr learned my addr 172.21.15.5:0/3210012480 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): get_auth_request con 0x7fa608006720 auth_method 0 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:17.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa6080b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6163cc700 1 -- 172.21.15.5:0/3210012480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 msgr2=0x7fa6080b7e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6163cc700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 0x7fa6080b7e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): get_auth_request con 0x7fa608006b00 auth_method 0 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:17.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6153ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 -- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 msgr2=0x7fa6080b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa6080b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 -- 172.21.15.5:0/3210012480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 msgr2=0x7fa6080b7e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 0x7fa6080b7e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient: _finish_hunting 0 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 monclient: found mon.noname-a 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 -- 172.21.15.5:0/3210012480 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6000094b0 con 0x7fa608006720 2024-09-16T10:50:17.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 client.?.objecter ms_handle_connect 0x7fa608006720 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa615bcb700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00b7b0 tx=0x7fa60c00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c003e70 con 0x7fa608006720 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: dump: 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:17.431 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 5 monclient: authenticate success, global_id 4467 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c005070 con 0x7fa608006720 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _renew_subs 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60801cb10 con 0x7fa608006720 2024-09-16T10:50:17.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c005420 con 0x7fa608006720 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa6067fc700 10 monclient: dump: 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:17.433 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _renew_subs 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa607fff700 4 set_mon_vals no callback set 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60801dd50 con 0x7fa608006720 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.412+0000 7fa616bcd700 1 librados: init done 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fa60c005580 con 0x7fa608006720 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa616bcd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa616bcd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa616bcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608021060 con 0x7fa608006720 2024-09-16T10:50:17.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 1 --2- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 0x7fa5f40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7fa60c0470d0 con 0x7fa608006720 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 10 client.4467.objecter ms_dispatch 0x7fa610000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 3 client.4467.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 3 client.4467.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6153ca700 1 --2- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 0x7fa5f40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6153ca700 10 monclient: get_auth_request con 0x7fa5f4036f10 auth_method 0 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 20 client.4467.objecter dump_active .. 0 homeless 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6153ca700 1 --2- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 0x7fa5f40393d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa600009850 tx=0x7fa600008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.416+0000 7fa6067fc700 10 client.4467.objecter ms_handle_connect 0x7fa5f4036f10 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.420+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa60c005830 con 0x7fa608006720 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.420+0000 7fa6067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.420+0000 7fa6067fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:17.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa60801c890 con 0x7fa608006720 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa6067fc700 1 -- 172.21.15.5:0/3210012480 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fa60c019360 con 0x7fa608006720 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa6067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:17.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa6067fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:17.661 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:50:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 librados: watch_flush enter 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 librados: watch_flush exit 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 20 client.4467.objecter shutdown clearing up homeless session... 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 client.4467.objecter successfully canceled tick 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 msgr2=0x7fa5f40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 0x7fa5f40393d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa600009850 tx=0x7fa600008030 comp rx=0 tx=0).stop 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 10 monclient: shutdown 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 msgr2=0x7fa608150410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00b7b0 tx=0x7fa60c00bac0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 shutdown_connections 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6080b8490 0x7fa6080b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa608006b00 0x7fa6080b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa5f4036f10 0x7fa5f40393d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 --2- 172.21.15.5:0/3210012480 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa608006720 0x7fa608150410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 >> 172.21.15.5:0/3210012480 conn(0x7fa608027320 msgr2=0x7fa608028710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 shutdown_connections 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.656+0000 7fa616bcd700 1 -- 172.21.15.5:0/3210012480 wait complete. 2024-09-16T10:50:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:17.660+0000 7fa616bcd700 1 librados: shutdown 2024-09-16T10:50:20.676 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: get_monmap_and_config 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: monmap: 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:20.793735+0000 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:20.793735+0000 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:20.798 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 Processor -- start 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 -- start start 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: init 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:20.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0ee00a2c80 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0ee00a35a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0ee0135f80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee01368d0 con 0x7f0ee00a2c80 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00a3db0 con 0x7f0ee00a35a0 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00a4b70 con 0x7f0ee0135f80 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:20.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0edf59e700 10 monclient(hunting): authenticate will time out at 879.370092s 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eddd9b700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 msgr2=0x7f0ee0135a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eddd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60894/0 (socket says 172.21.15.5:60894) 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 learned_addr learned my addr 172.21.15.5:0/1527407095 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ee0135f80 auth_method 0 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 10 monclient(hunting): get_auth_request con 0x7f0ee00a2c80 auth_method 0 2024-09-16T10:50:20.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:20.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.792+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:21.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 msgr2=0x7f0ee0135a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:21.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:50:21.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:21.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 10 monclient(hunting): get_auth_request con 0x7f0ee00a2c80 auth_method 0 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ee0135f80 auth_method 0 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:21.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:50:21.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:50:21.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:21.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:21.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:20.996+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:50:21.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 msgr2=0x7f0ee0135a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:21.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:50:21.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:21.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:21.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 10 monclient(hunting): get_auth_request con 0x7f0ee00a2c80 auth_method 0 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ee0135f80 auth_method 0 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:21.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:21.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:50:21.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:21.400+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:50:22.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 msgr2=0x7f0ee0135a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:22.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:50:22.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:22.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ee0135f80 auth_method 0 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 10 monclient(hunting): get_auth_request con 0x7f0ee00a2c80 auth_method 0 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:50:22.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:50:22.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:50:22.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:50:22.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:22.204+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:50:23.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): tick 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 monclient(hunting): continuing hunt 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 msgr2=0x7f0ee0136390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ee0135f80 0x7f0ee0136390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 msgr2=0x7f0ee0135a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.143:3300/0 conn(0x7f0ee00a35a0 0x7f0ee0135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 -- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 msgr2=0x7f0ee00a3060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:23.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 --2- 172.21.15.5:0/1527407095 >> v2:172.21.15.5:3300/0 conn(0x7f0ee00a2c80 0x7f0ee00a3060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient: _add_conns ranks=[0,1] 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0003740 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0007470 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 -- 172.21.15.5:0/1527407095 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee00a0850 con 0x7f0ec0003740 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 1 -- 172.21.15.5:0/1527407095 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec000c1b0 con 0x7f0ec0007470 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0ecffff700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:23.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.792+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v1:172.21.15.143:6789/0 conn(0x7f0ec0007470 legacy=0x7f0ec0009930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:23.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:23.996+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v1:172.21.15.143:6789/0 conn(0x7f0ec0007470 legacy=0x7f0ec0009930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:24.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.400+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> v1:172.21.15.143:6789/0 conn(0x7f0ec0007470 legacy=0x7f0ec0009930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:24.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616089494 0 0) 0x7f0ee00a0850 con 0x7f0ec0003740 2024-09-16T10:50:24.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 10 monclient(hunting): _negotiate 2024-09-16T10:50:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 10 monclient(hunting): my global_id is 4470 2024-09-16T10:50:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.612+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec80035d0 con 0x7f0ec0003740 2024-09-16T10:50:24.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 276359771 0 0) 0x7f0ec80035d0 con 0x7f0ec0003740 2024-09-16T10:50:24.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 >> v1:172.21.15.143:6789/0 conn(0x7f0ec0007470 legacy=0x7f0ec0009930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _finish_hunting 0 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec000d3b0 con 0x7f0ec0003740 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _finish_auth 0 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _check_auth_tickets 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f0ed400c570 con 0x7f0ec0003740 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edf59e700 5 monclient: authenticate success, global_id 4470 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:24.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: dump: 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 >> v1:172.21.15.5:6789/0 conn(0x7f0ec0003740 legacy=0x7f0ec0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 0x7f0ec8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 1 -- 172.21.15.5:0/1527407095 reap_dead start 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): picked mon.b con 0x7f0ec8003a50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 0x7f0ec8004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): picked mon.c con 0x7f0ec8004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8005350 0x7f0ec800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): picked mon.a con 0x7f0ec8005350 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec800c240 con 0x7f0ec8005350 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec800c3b0 con 0x7f0ec8003a50 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec800c570 con 0x7f0ec8004870 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edd59a700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0ede59c700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 msgr2=0x7f0ec8003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0ede59c700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 0x7f0ec8003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0edf59e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 0x7f0ec8004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): get_auth_request con 0x7f0ec8004870 auth_method 0 2024-09-16T10:50:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8005350 0x7f0ec800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ec8005350 auth_method 0 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient(hunting): handle_auth_done global_id 4470 payload 293 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 msgr2=0x7f0ec8003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 0x7f0ec8003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8005350 msgr2=0x7f0ec800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8005350 0x7f0ec800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient: _finish_hunting 0 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 monclient: found mon.c 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 -- 172.21.15.5:0/1527407095 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec80035d0 con 0x7f0ec8004870 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient: _finish_auth 0 2024-09-16T10:50:24.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 10 monclient: _check_auth_tickets 2024-09-16T10:50:24.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:24.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.616+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 0x7f0ec8004cf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007fd0 tx=0x7f0ec80058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1f90894e1648469 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0007bf0 con 0x7f0ec8004870 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: dump: 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:24.637 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed000cdd0 con 0x7f0ec8004870 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 4 set_mon_vals no callback set 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 1 -- 172.21.15.5:0/1527407095 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0012640 con 0x7f0ec8004870 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:24.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edd59a700 10 monclient: dump: 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 10 monclient: shutdown 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:24.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 -- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 msgr2=0x7f0ec8004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 0x7f0ec8004cf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0007fd0 tx=0x7f0ec80058a0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 -- 172.21.15.5:0/1527407095 shutdown_connections 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8005350 0x7f0ec800ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8004870 0x7f0ec8004cf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 --2- 172.21.15.5:0/1527407095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8003a50 0x7f0ec8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 -- 172.21.15.5:0/1527407095 >> 172.21.15.5:0/1527407095 conn(0x7f0ee00a4d80 msgr2=0x7f0ee00a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 -- 172.21.15.5:0/1527407095 shutdown_connections 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.620+0000 7f0edf59e700 1 -- 172.21.15.5:0/1527407095 wait complete. 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: build_initial_monmap 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: monmap: 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:24.640 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 librados: starting msgr at 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 librados: starting objecter 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 Processor -- start 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- start start 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 librados: setting wanted keys 2024-09-16T10:50:24.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 librados: calling monclient init 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: init 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0ec8003a50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 0x7f0ee01ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0ec8004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8005350 0x7f0ee0122080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0ec8005350 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0125d00 con 0x7f0ec8003a50 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0125fe0 con 0x7f0ec8004870 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0124190 con 0x7f0ec8005350 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient(hunting): authenticate will time out at 883.202394s 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44462/0 (socket says 172.21.15.5:44462) 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 -- 172.21.15.5:0/3975459245 learned_addr learned my addr 172.21.15.5:0/3975459245 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): get_auth_request con 0x7f0ec8003a50 auth_method 0 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:24.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eddd9b700 1 -- 172.21.15.5:0/3975459245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 msgr2=0x7f0ee01ca940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 0x7f0ee01ca940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8005350 0x7f0ee0122080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): get_auth_request con 0x7f0ec8005350 auth_method 0 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:24.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 -- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8005350 msgr2=0x7f0ee0122080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8005350 0x7f0ee0122080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 -- 172.21.15.5:0/3975459245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 msgr2=0x7f0ee01ca940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 0x7f0ee01ca940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient: _finish_hunting 0 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 monclient: found mon.noname-a 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 -- 172.21.15.5:0/3975459245 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee0124850 con 0x7f0ec8003a50 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0ede59c700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4012950 tx=0x7f0ed4012c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:24.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 client.?.objecter ms_handle_connect 0x7f0ec8003a50 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4010ad0 con 0x7f0ec8003a50 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: dump: 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:24.646 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: _finish_auth 0 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: _check_auth_tickets 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed400c040 con 0x7f0ec8003a50 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 5 monclient: authenticate success, global_id 4473 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _renew_subs 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee0135ad0 con 0x7f0ec8003a50 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4010e70 con 0x7f0ec8003a50 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:24.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0eceffd700 10 monclient: dump: 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:24.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edcd99700 4 set_mon_vals no callback set 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _renew_subs 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee0123c80 con 0x7f0ec8003a50 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.624+0000 7f0edf59e700 1 librados: init done 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0ed40153f0 con 0x7f0ec8003a50 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0edf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0edf59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0edf59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee009f0d0 con 0x7f0ec8003a50 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 1 --2- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 0x7f0ec0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:24.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f0ed4000e40 con 0x7f0ec8003a50 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 10 client.4473.objecter ms_dispatch 0x7f0ee0000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 3 client.4473.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 3 client.4473.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 20 client.4473.objecter dump_active .. 0 homeless 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 0x7f0ec0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eddd9b700 10 monclient: get_auth_request con 0x7f0ec0037810 auth_method 0 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eceffd700 10 client.4473.objecter ms_handle_connect 0x7f0ec0037810 2024-09-16T10:50:24.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.628+0000 7f0eddd9b700 1 --2- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 0x7f0ec0039cd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0123880 tx=0x7f0ed0010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.632+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0ed401ed50 con 0x7f0ec8003a50 2024-09-16T10:50:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.632+0000 7f0eceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.632+0000 7f0eceffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0edf59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:50:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0edf59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0edf59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:24.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ee00721d0 con 0x7f0ec8003a50 2024-09-16T10:50:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0eceffd700 1 -- 172.21.15.5:0/3975459245 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f0ed401e310 con 0x7f0ec8003a50 2024-09-16T10:50:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0eceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:50:24.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.868+0000 7f0eceffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:24.873 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:50:24.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 10 librados: watch_flush enter 2024-09-16T10:50:24.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 10 librados: watch_flush exit 2024-09-16T10:50:24.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 20 client.4473.objecter shutdown clearing up homeless session... 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 10 client.4473.objecter successfully canceled tick 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 msgr2=0x7f0ec0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 0x7f0ec0039cd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0123880 tx=0x7f0ed0010000 comp rx=0 tx=0).stop 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 10 monclient: shutdown 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 msgr2=0x7f0ee01ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4012950 tx=0x7f0ed4012c60 comp rx=0 tx=0).stop 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 shutdown_connections 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0ec8005350 0x7f0ee0122080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ec8004870 0x7f0ee01ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ec0037810 0x7f0ec0039cd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 --2- 172.21.15.5:0/3975459245 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ec8003a50 0x7f0ee01ca400 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:24.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 >> 172.21.15.5:0/3975459245 conn(0x7f0ee00a4d80 msgr2=0x7f0ee00a6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:24.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 shutdown_connections 2024-09-16T10:50:24.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 -- 172.21.15.5:0/3975459245 wait complete. 2024-09-16T10:50:24.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:24.872+0000 7f0edf59e700 1 librados: shutdown 2024-09-16T10:50:24.894 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T10:50:24.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T10:50:24.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 10 monclient: get_monmap_and_config 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 10 monclient: build_initial_monmap 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 10 monclient: monmap: 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.006 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:25.003946+0000 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:25.003946+0000 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:25.007 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 1 Processor -- start 2024-09-16T10:50:25.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 1 -- start start 2024-09-16T10:50:25.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.000+0000 7f4b748de700 10 monclient: init 2024-09-16T10:50:25.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b700a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-a con 0x7f4b7009f780 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-b con 0x7f4b700a36e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700a3c40 con 0x7f4b700a2bf0 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700a4300 con 0x7f4b7009f780 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7009fda0 con 0x7f4b700a36e0 2024-09-16T10:50:25.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient(hunting): authenticate will time out at 883.580493s 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6f7fe700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f4b700a2bf0 legacy=0x7f4b700a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:50170/0 (socket says 172.21.15.5:50170) 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44474/0 (socket says 172.21.15.5:44474) 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 -- 172.21.15.5:0/4104652563 learned_addr learned my addr 172.21.15.5:0/4104652563 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): get_auth_request con 0x7f4b700a36e0 auth_method 0 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): get_auth_request con 0x7f4b7009f780 auth_method 0 2024-09-16T10:50:25.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 -- 172.21.15.5:0/4104652563 >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 msgr2=0x7f4b7013ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 --2- 172.21.15.5:0/4104652563 >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 -- 172.21.15.5:0/4104652563 >> v1:172.21.15.143:6789/0 conn(0x7f4b700a2bf0 legacy=0x7f4b700a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 monclient: found mon.noname-b 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 -- 172.21.15.5:0/4104652563 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700a0080 con 0x7f4b700a36e0 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6effd700 1 --2- 172.21.15.5:0/4104652563 >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6ffff700 1 --2- 172.21.15.5:0/4104652563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009e50 tx=0x7f4b6800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89ed56bb51ff1c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 1 -- 172.21.15.5:0/4104652563 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800e070 con 0x7f4b700a36e0 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: dump: 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.015 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 1 -- 172.21.15.5:0/4104652563 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68003a40 con 0x7f4b700a36e0 2024-09-16T10:50:25.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 5 monclient: authenticate success, global_id 4394 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 4 set_mon_vals no callback set 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 1 -- 172.21.15.5:0/4104652563 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68003df0 con 0x7f4b700a36e0 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b6e7fc700 10 monclient: dump: 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.017 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 10 monclient: shutdown 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- 172.21.15.5:0/4104652563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 msgr2=0x7f4b7013d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 --2- 172.21.15.5:0/4104652563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009e50 tx=0x7f4b6800b330 comp rx=0 tx=0).stop 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- 172.21.15.5:0/4104652563 shutdown_connections 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 --2- 172.21.15.5:0/4104652563 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a36e0 0x7f4b7013d350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 --2- 172.21.15.5:0/4104652563 >> v2:172.21.15.5:3300/0 conn(0x7f4b7009f780 0x7f4b7013ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- 172.21.15.5:0/4104652563 >> 172.21.15.5:0/4104652563 conn(0x7f4b700a53e0 msgr2=0x7f4b70073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- 172.21.15.5:0/4104652563 shutdown_connections 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.004+0000 7f4b748de700 1 -- 172.21.15.5:0/4104652563 wait complete. 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: build_initial_monmap 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: monmap: 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:25.019 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 librados: starting msgr at 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 librados: starting objecter 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 Processor -- start 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- start start 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 librados: setting wanted keys 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 librados: calling monclient init 2024-09-16T10:50:25.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: init 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7009f780 0x7f4b701359a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-a con 0x7f4b7009f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 0x7f4b7013aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-c con 0x7f4b700a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b700a36e0 0x7f4b7013b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): picked mon.noname-b con 0x7f4b700a36e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701200e0 con 0x7f4b7009f780 2024-09-16T10:50:25.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70120330 con 0x7f4b700a36e0 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700a7390 con 0x7f4b700a2bf0 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient(hunting): authenticate will time out at 883.585602s 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 0x7f4b7013aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6ffff700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b700a36e0 msgr2=0x7f4b7013b0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7009f780 0x7f4b701359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b700a36e0 0x7f4b7013b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 0x7f4b7013aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:50180/0 (socket says 172.21.15.5:50180) 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 -- 172.21.15.5:0/2833478717 learned_addr learned my addr 172.21.15.5:0/2833478717 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): get_auth_request con 0x7f4b700a2bf0 auth_method 0 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): get_auth_request con 0x7f4b7009f780 auth_method 0 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b700a36e0 msgr2=0x7f4b7013b0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b700a36e0 0x7f4b7013b0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7009f780 msgr2=0x7f4b701359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7009f780 0x7f4b701359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient: _finish_hunting 0 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 monclient: found mon.noname-c 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b580094b0 con 0x7f4b700a2bf0 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6f7fe700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7009f780 0x7f4b701359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:50:25.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 client.?.objecter ms_handle_connect 0x7f4b700a2bf0 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6effd700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 0x7f4b7013aba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800ec40 tx=0x7f4b5800ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800c910 con 0x7f4b700a2bf0 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: dump: 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: _finish_auth 0 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: _check_auth_tickets 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 5 monclient: authenticate success, global_id 4397 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5800ca70 con 0x7f4b700a2bf0 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _renew_subs 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:25.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b68009b30 con 0x7f4b700a2bf0 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5800ce20 con 0x7f4b700a2bf0 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b6dffb700 4 set_mon_vals no callback set 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b67fff700 10 monclient: dump: 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.032 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _renew_subs 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7009f4e0 con 0x7f4b700a2bf0 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.008+0000 7f4b748de700 1 librados: init done 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b748de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b748de700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 0x7f4b7013ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701204c0 con 0x7f4b7013b620 2024-09-16T10:50:25.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 0x7f4b7013ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient: get_auth_request con 0x7f4b7013b620 auth_method 0 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4b58013470 con 0x7f4b700a2bf0 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b7013d890 con 0x7f4b7013b620 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 --2- 172.21.15.5:0/2833478717 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4b50036f10 0x7f4b500393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6f7fe700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 0x7f4b7013ba00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4b600055f0 tx=0x7f4b600059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 client.4397.objecter ms_handle_connect 0x7f4b7013b620 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 client.4397.objecter resend_mon_ops 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f4b58047400 con 0x7f4b700a2bf0 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 client.4397.objecter ms_dispatch 0x7f4b70000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 3 client.4397.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 3 client.4397.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 20 client.4397.objecter dump_active .. 0 homeless 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000e830 con 0x7f4b7013b620 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 monclient: dump: 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.035 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6ffff700 1 --2- 172.21.15.5:0/2833478717 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4b50036f10 0x7f4b500393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6ffff700 10 monclient: get_auth_request con 0x7f4b50036f10 auth_method 0 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f4b6000e990 con 0x7f4b7013b620 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:25.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 msgr2=0x7f4b7013ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 0x7f4b7013ba00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4b600055f0 tx=0x7f4b600059b0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 msgr2=0x7f4b7013ba00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:25.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b7013b620 0x7f4b7013ba00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b67fff700 10 client.4397.objecter ms_handle_connect 0x7f4b50036f10 2024-09-16T10:50:25.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.012+0000 7f4b6ffff700 1 --2- 172.21.15.5:0/2833478717 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4b50036f10 0x7f4b500393d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b7013cfe0 tx=0x7f4b68009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 0x7f4b701371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70074d50 con 0x7f4b70136de0 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 0x7f4b701371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient: get_auth_request con 0x7f4b70136de0 auth_method 0 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 10 monclient(hunting): handle_auth_done global_id 4485 payload 824 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 1 -- 172.21.15.5:0/2833478717 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4b7013bf40 con 0x7f4b70136de0 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6f7fe700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 0x7f4b701371f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b60018000 tx=0x7f4b60005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 client.4397.objecter ms_handle_connect 0x7f4b70136de0 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 client.4397.objecter resend_mon_ops 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60006030 con 0x7f4b70136de0 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 monclient: dump: 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7f4b60006190 con 0x7f4b70136de0 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 msgr2=0x7f4b701371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 0x7f4b701371f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4b60018000 tx=0x7f4b60005a80 comp rx=0 tx=0).stop 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 msgr2=0x7f4b701371f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b67fff700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b70136de0 0x7f4b701371f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 16, 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:50:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.043 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.044 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:50:25.045 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.046 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.047 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.048 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.049 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.051 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.052 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:50:25.053 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:50:25.054 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 librados: watch_flush enter 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 librados: watch_flush exit 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 20 client.4397.objecter shutdown clearing up homeless session... 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 client.4397.objecter successfully canceled tick 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4b50036f10 msgr2=0x7f4b500393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 --2- 172.21.15.5:0/2833478717 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4b50036f10 0x7f4b500393d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b7013cfe0 tx=0x7f4b68009000 comp rx=0 tx=0).stop 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 10 monclient: shutdown 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 msgr2=0x7f4b7013aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 --2- 172.21.15.5:0/2833478717 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4b700a2bf0 0x7f4b7013aba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800ec40 tx=0x7f4b5800ef50 comp rx=0 tx=0).stop 2024-09-16T10:50:25.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6ffff700 1 -- 172.21.15.5:0/2833478717 reap_dead start 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b6ffff700 1 -- 172.21.15.5:0/2833478717 reap_dead start 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 shutdown_connections 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 >> 172.21.15.5:0/2833478717 conn(0x7f4b700a53e0 msgr2=0x7f4b7006eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 shutdown_connections 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 -- 172.21.15.5:0/2833478717 wait complete. 2024-09-16T10:50:25.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.024+0000 7f4b748de700 1 librados: shutdown 2024-09-16T10:50:25.057 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:50:25.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T10:50:25.057 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:50:25.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: get_monmap_and_config 2024-09-16T10:50:25.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: build_initial_monmap 2024-09-16T10:50:25.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: monmap: 2024-09-16T10:50:25.179 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:50:25.177164+0000 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:50:25.177164+0000 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:50:25.180 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 Processor -- start 2024-09-16T10:50:25.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- start start 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: init 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 0x7f1f7009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-c con 0x7f1f7009e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1f7009d6a0 0x7f1f7009f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-a con 0x7f1f7009d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f7009dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f70070e40 con 0x7f1f7009dfd0 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700a01d0 con 0x7f1f7009d6a0 2024-09-16T10:50:25.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7009f740 con 0x7f1f7009e8e0 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient(hunting): authenticate will time out at 883.753539s 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f758a7700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 msgr2=0x7f1f7009ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f758a7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 0x7f1f7009ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1f7009d6a0 0x7f1f7009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f760a8700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1f7009dfd0 0x7f1f701336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60724/0 (socket says 172.21.15.5:60724) 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f760a8700 1 -- 172.21.15.5:0/797745932 learned_addr learned my addr 172.21.15.5:0/797745932 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): get_auth_request con 0x7f1f7009d6a0 auth_method 0 2024-09-16T10:50:25.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 -- 172.21.15.5:0/797745932 >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 msgr2=0x7f1f7009ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 --2- 172.21.15.5:0/797745932 >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 0x7f1f7009ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 -- 172.21.15.5:0/797745932 >> v1:172.21.15.5:6789/0 conn(0x7f1f7009dfd0 legacy=0x7f1f701336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient: _finish_hunting 0 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 monclient: found mon.noname-a 2024-09-16T10:50:25.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 -- 172.21.15.5:0/797745932 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7009fef0 con 0x7f1f7009d6a0 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f750a6700 1 --2- 172.21.15.5:0/797745932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009d6a0 0x7f1f7009f200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009850 tx=0x7f1f6c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7edad97caa45e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 1 -- 172.21.15.5:0/797745932 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c011830 con 0x7f1f7009d6a0 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: dump: 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.192 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: _finish_auth 0 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: _check_auth_tickets 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 5 monclient: authenticate success, global_id 4488 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 1 -- 172.21.15.5:0/797745932 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c011990 con 0x7f1f7009d6a0 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: get_monmap_and_config success 2024-09-16T10:50:25.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 4 set_mon_vals no callback set 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 1 -- 172.21.15.5:0/797745932 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c011d20 con 0x7f1f7009d6a0 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f748a5700 10 monclient: dump: 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.194 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.195 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 10 monclient: shutdown 2024-09-16T10:50:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- 172.21.15.5:0/797745932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009d6a0 msgr2=0x7f1f7009f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 --2- 172.21.15.5:0/797745932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009d6a0 0x7f1f7009f200 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009850 tx=0x7f1f6c009b60 comp rx=0 tx=0).stop 2024-09-16T10:50:25.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- 172.21.15.5:0/797745932 shutdown_connections 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 --2- 172.21.15.5:0/797745932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009d6a0 0x7f1f7009f200 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 --2- 172.21.15.5:0/797745932 >> v2:172.21.15.143:3300/0 conn(0x7f1f7009e8e0 0x7f1f7009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- 172.21.15.5:0/797745932 >> 172.21.15.5:0/797745932 conn(0x7f1f700a45d0 msgr2=0x7f1f700a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- 172.21.15.5:0/797745932 shutdown_connections 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.176+0000 7f1f768a9700 1 -- 172.21.15.5:0/797745932 wait complete. 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: build_initial_monmap 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: monmap: 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:50:25.197 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 librados: starting msgr at 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 librados: starting objecter 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 Processor -- start 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- start start 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 librados: setting wanted keys 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 librados: calling monclient init 2024-09-16T10:50:25.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: init 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _reopen_session rank -1 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7009d6a0 0x7f1f701cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-c con 0x7f1f7009d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 0x7f1f701cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-a con 0x7f1f7009dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7009e8e0 0x7f1f701220c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): picked mon.noname-b con 0x7f1f7009e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700a3860 con 0x7f1f7009dfd0 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7009b0d0 con 0x7f1f7009e8e0 2024-09-16T10:50:25.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70123d60 con 0x7f1f7009d6a0 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): _renew_subs 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): authenticate will time out at 883.757329s 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 0x7f1f701cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 0x7f1f701cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44524/0 (socket says 172.21.15.5:44524) 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 -- 172.21.15.5:0/1432325486 learned_addr learned my addr 172.21.15.5:0/1432325486 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): get_auth_request con 0x7f1f7009dfd0 auth_method 0 2024-09-16T10:50:25.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f760a8700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7009e8e0 msgr2=0x7f1f701220c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f760a8700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7009e8e0 0x7f1f701220c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7009d6a0 0x7f1f701cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): get_auth_request con 0x7f1f7009d6a0 auth_method 0 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7009d6a0 msgr2=0x7f1f701cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f7009d6a0 0x7f1f701cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7009e8e0 msgr2=0x7f1f701220c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f7009e8e0 0x7f1f701220c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient: _finish_hunting 0 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 monclient: found mon.noname-a 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:50:25.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0094b0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f750a6700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 0x7f1f701cb130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 client.?.objecter ms_handle_connect 0x7f1f7009dfd0 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0044f0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: dump: 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.203 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: _finish_auth 0 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: _check_auth_tickets 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:50:25.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 5 monclient: authenticate success, global_id 4494 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _renew_subs 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f701246b0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c01f070 con 0x7f1f7009dfd0 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _renew_subs 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f701260c0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 librados: init done 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f67fff700 4 set_mon_vals no callback set 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0038c0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 10 monclient: dump: 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.206 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:25.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1f6c021030 con 0x7f1f7009dfd0 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 0x7f1f701229e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70067380 con 0x7f1f70122600 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.180+0000 7f1f758a7700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 0x7f1f701229e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient: get_auth_request con 0x7f1f70122600 auth_method 0 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f70134160 con 0x7f1f70122600 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 --2- 172.21.15.5:0/1432325486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f54036f10 0x7f1f540393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f1f6c048df0 con 0x7f1f7009dfd0 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f758a7700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 0x7f1f701229e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009710 tx=0x7f1f60009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 client.4494.objecter ms_dispatch 0x7f1f70000fd0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 3 client.4494.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 3 client.4494.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 20 client.4494.objecter dump_active .. 0 homeless 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 client.4494.objecter ms_handle_connect 0x7f1f70122600 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 client.4494.objecter resend_mon_ops 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60012600 con 0x7f1f70122600 2024-09-16T10:50:25.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f760a8700 1 --2- 172.21.15.5:0/1432325486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f54036f10 0x7f1f540393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 monclient: dump: 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.210 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f760a8700 10 monclient: get_auth_request con 0x7f1f54036f10 auth_method 0 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f1f60012760 con 0x7f1f70122600 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 msgr2=0x7f1f701229e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 0x7f1f701229e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009710 tx=0x7f1f60009ad0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 msgr2=0x7f1f701229e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122600 0x7f1f701229e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f760a8700 1 --2- 172.21.15.5:0/1432325486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f54036f10 0x7f1f540393d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f701331c0 tx=0x7f1f680015d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.184+0000 7f1f667fc700 10 client.4494.objecter ms_handle_connect 0x7f1f54036f10 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 0x7f1f7006e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700008d0 con 0x7f1f70122f20 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 0x7f1f7006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient: get_auth_request con 0x7f1f70122f20 auth_method 0 2024-09-16T10:50:25.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 1 -- 172.21.15.5:0/1432325486 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1f70133cc0 con 0x7f1f70122f20 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 client.4494.objecter ms_handle_connect 0x7f1f70122f20 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 client.4494.objecter resend_mon_ops 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f758a7700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 0x7f1f7006e580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009710 tx=0x7f1f60005a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60023070 con 0x7f1f70122f20 2024-09-16T10:50:25.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 monclient: dump: 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7f1f60012e10 con 0x7f1f70122f20 2024-09-16T10:50:25.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 msgr2=0x7f1f7006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 0x7f1f7006e580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009710 tx=0x7f1f60005a80 comp rx=0 tx=0).stop 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 msgr2=0x7f1f7006e580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f667fc700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f70122f20 0x7f1f7006e580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:25.216 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 16, 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.217 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.218 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:50:25.219 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:50:25.220 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:50:25.221 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:50:25.222 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.224 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:50:25.225 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.226 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.227 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.228 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:50:25.229 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:50:25.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:50:25.230 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:50:25.230 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:50:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 librados: watch_flush enter 2024-09-16T10:50:25.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 librados: watch_flush exit 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 20 client.4494.objecter shutdown clearing up homeless session... 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 client.4494.objecter successfully canceled tick 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f54036f10 msgr2=0x7f1f540393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 --2- 172.21.15.5:0/1432325486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f54036f10 0x7f1f540393d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1f701331c0 tx=0x7f1f680015d0 comp rx=0 tx=0).stop 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 10 monclient: shutdown 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 msgr2=0x7f1f701cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 --2- 172.21.15.5:0/1432325486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f7009dfd0 0x7f1f701cb130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004770 comp rx=0 tx=0).stop 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f760a8700 1 -- 172.21.15.5:0/1432325486 reap_dead start 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f760a8700 1 -- 172.21.15.5:0/1432325486 reap_dead start 2024-09-16T10:50:25.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 shutdown_connections 2024-09-16T10:50:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 >> 172.21.15.5:0/1432325486 conn(0x7f1f700a45d0 msgr2=0x7f1f7009b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:50:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 shutdown_connections 2024-09-16T10:50:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 -- 172.21.15.5:0/1432325486 wait complete. 2024-09-16T10:50:25.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:50:25.192+0000 7f1f768a9700 1 librados: shutdown 2024-09-16T10:50:25.233 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:50:25.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:51:06.887 INFO:tasks.workunit.client.0.smithi143.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T10:51:06.887 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.887 INFO:tasks.workunit.client.0.smithi143.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T10:51:06.887 INFO:tasks.workunit.client.0.smithi143.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: git switch -c 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr:Or undo this operation with: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: git switch - 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.888 INFO:tasks.workunit.client.0.smithi143.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T10:51:06.889 INFO:tasks.workunit.client.0.smithi143.stderr: 2024-09-16T10:51:06.889 INFO:tasks.workunit.client.0.smithi143.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T10:51:06.892 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T10:51:06.925 INFO:tasks.workunit.client.0.smithi143.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T10:51:06.928 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T10:51:06.928 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T10:51:07.814 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T10:51:07.902 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T10:51:07.957 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T10:51:07.960 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T10:51:07.960 INFO:tasks.workunit.client.0.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T10:51:08.020 INFO:tasks.workunit.client.0.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T10:51:08.025 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T10:51:08.025 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T10:51:08.076 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T10:51:08.077 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T10:51:08.078 DEBUG:teuthology.orchestra.run.smithi143:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T10:51:08.093 INFO:tasks.workunit.client.0.smithi143.stderr:+ parallel=1 2024-09-16T10:51:08.093 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' '' = --serial ']' 2024-09-16T10:51:08.093 INFO:tasks.workunit.client.0.smithi143.stderr:+ color= 2024-09-16T10:51:08.093 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -t 1 ']' 2024-09-16T10:51:08.094 INFO:tasks.workunit.client.0.smithi143.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T10:51:08.094 INFO:tasks.workunit.client.0.smithi143.stderr:+ declare -A pids 2024-09-16T10:51:08.094 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.094 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.095 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_aio 2024-09-16T10:51:08.095 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_aio' 2024-09-16T10:51:08.095 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_aio 2024-09-16T10:51:08.095 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.098 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_aio 2024-09-16T10:51:08.099 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24370 2024-09-16T10:51:08.099 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_aio on pid 24370' 2024-09-16T10:51:08.099 INFO:tasks.workunit.client.0.smithi143.stdout:test api_aio on pid 24370 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24370 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_aio_pp 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_aio_pp' 2024-09-16T10:51:08.100 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.101 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.101 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.101 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.101 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_aio 2024-09-16T10:51:08.102 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_aio_pp 2024-09-16T10:51:08.102 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T10:51:08.102 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.102 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T10:51:08.105 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_aio_pp 2024-09-16T10:51:08.105 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24378 2024-09-16T10:51:08.105 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_aio_pp on pid 24378' 2024-09-16T10:51:08.106 INFO:tasks.workunit.client.0.smithi143.stdout:test api_aio_pp on pid 24378 2024-09-16T10:51:08.106 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24378 2024-09-16T10:51:08.106 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.107 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.107 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T10:51:08.108 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_io 2024-09-16T10:51:08.108 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_io' 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_io 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.109 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_aio_pp 2024-09-16T10:51:08.110 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T10:51:08.110 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T10:51:08.111 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_io 2024-09-16T10:51:08.111 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24386 2024-09-16T10:51:08.112 INFO:tasks.workunit.client.0.smithi143.stdout:test api_io on pid 24386 2024-09-16T10:51:08.112 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_io on pid 24386' 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24386 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_io_pp 2024-09-16T10:51:08.113 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_io_pp' 2024-09-16T10:51:08.114 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_io_pp 2024-09-16T10:51:08.114 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.114 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.114 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.114 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.115 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.115 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T10:51:08.115 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_io 2024-09-16T10:51:08.116 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_io: /' 2024-09-16T10:51:08.117 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_io_pp 2024-09-16T10:51:08.117 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24394 2024-09-16T10:51:08.117 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_io_pp on pid 24394' 2024-09-16T10:51:08.118 INFO:tasks.workunit.client.0.smithi143.stdout:test api_io_pp on pid 24394 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24394 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_asio 2024-09-16T10:51:08.119 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_asio' 2024-09-16T10:51:08.120 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_asio 2024-09-16T10:51:08.120 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.121 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.121 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.121 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.121 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.121 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_io_pp 2024-09-16T10:51:08.122 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T10:51:08.122 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T10:51:08.122 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_asio 2024-09-16T10:51:08.122 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24402 2024-09-16T10:51:08.123 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_asio on pid 24402' 2024-09-16T10:51:08.123 INFO:tasks.workunit.client.0.smithi143.stdout:test api_asio on pid 24402 2024-09-16T10:51:08.125 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24402 2024-09-16T10:51:08.125 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.125 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T10:51:08.125 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.125 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_list 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_list' 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_list 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.126 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.127 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_asio 2024-09-16T10:51:08.127 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T10:51:08.127 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T10:51:08.127 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_list 2024-09-16T10:51:08.128 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24410 2024-09-16T10:51:08.128 INFO:tasks.workunit.client.0.smithi143.stdout:test api_list on pid 24410 2024-09-16T10:51:08.128 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_list on pid 24410' 2024-09-16T10:51:08.129 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24410 2024-09-16T10:51:08.129 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.129 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.129 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_lock 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_lock' 2024-09-16T10:51:08.130 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T10:51:08.132 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_list 2024-09-16T10:51:08.132 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_lock 2024-09-16T10:51:08.132 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.132 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_list: /' 2024-09-16T10:51:08.135 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_lock 2024-09-16T10:51:08.136 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24418 2024-09-16T10:51:08.136 INFO:tasks.workunit.client.0.smithi143.stdout:test api_lock on pid 24418 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_lock on pid 24418' 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24418 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_lock_pp 2024-09-16T10:51:08.137 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_lock_pp' 2024-09-16T10:51:08.138 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.138 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.138 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.138 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.139 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_lock_pp 2024-09-16T10:51:08.139 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.139 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_lock 2024-09-16T10:51:08.141 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stdout:test api_lock_pp on pid 24426 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_lock_pp 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24426 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_lock_pp on pid 24426' 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24426 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.144 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.145 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T10:51:08.145 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_misc 2024-09-16T10:51:08.145 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_misc' 2024-09-16T10:51:08.146 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_misc 2024-09-16T10:51:08.146 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.146 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.147 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T10:51:08.149 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.149 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.149 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.150 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_lock_pp 2024-09-16T10:51:08.150 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T10:51:08.150 INFO:tasks.workunit.client.0.smithi143.stdout:test api_misc on pid 24441 2024-09-16T10:51:08.150 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_misc 2024-09-16T10:51:08.150 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24441 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_misc on pid 24441' 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24441 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_misc_pp 2024-09-16T10:51:08.151 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_misc_pp' 2024-09-16T10:51:08.152 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T10:51:08.152 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T10:51:08.157 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_misc_pp 2024-09-16T10:51:08.158 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.161 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_misc_pp 2024-09-16T10:51:08.161 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24477 2024-09-16T10:51:08.161 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_misc_pp on pid 24477' 2024-09-16T10:51:08.161 INFO:tasks.workunit.client.0.smithi143.stdout:test api_misc_pp on pid 24477 2024-09-16T10:51:08.162 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24477 2024-09-16T10:51:08.162 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_tier_pp 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_tier_pp' 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.163 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.164 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.164 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.164 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.164 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.164 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_misc 2024-09-16T10:51:08.169 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_misc_pp 2024-09-16T10:51:08.169 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T10:51:08.170 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T10:51:08.173 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_tier_pp 2024-09-16T10:51:08.173 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.177 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T10:51:08.178 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T10:51:08.179 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_tier_pp 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stdout:test api_tier_pp on pid 24513 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24513 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_tier_pp on pid 24513' 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24513 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.180 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.181 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T10:51:08.181 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_pool 2024-09-16T10:51:08.182 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_pool' 2024-09-16T10:51:08.183 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.184 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_pool 2024-09-16T10:51:08.184 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.184 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.184 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.184 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.185 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_pool 2024-09-16T10:51:08.185 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_tier_pp 2024-09-16T10:51:08.185 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24534 2024-09-16T10:51:08.185 INFO:tasks.workunit.client.0.smithi143.stdout:test api_pool on pid 24534 2024-09-16T10:51:08.186 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_pool on pid 24534' 2024-09-16T10:51:08.186 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24534 2024-09-16T10:51:08.186 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.186 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T10:51:08.186 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.187 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_snapshots 2024-09-16T10:51:08.187 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_snapshots' 2024-09-16T10:51:08.187 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T10:51:08.187 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.187 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_snapshots 2024-09-16T10:51:08.188 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T10:51:08.189 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.189 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.189 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.189 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.189 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_pool 2024-09-16T10:51:08.190 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T10:51:08.191 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T10:51:08.191 INFO:tasks.workunit.client.0.smithi143.stdout:test api_snapshots on pid 24561 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_snapshots 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24561 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_snapshots on pid 24561' 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24561 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.192 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T10:51:08.193 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_snapshots_pp 2024-09-16T10:51:08.193 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_snapshots_pp' 2024-09-16T10:51:08.193 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.193 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_snapshots_pp 2024-09-16T10:51:08.194 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.194 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.194 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.194 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.196 INFO:tasks.workunit.client.0.smithi143.stdout:test api_snapshots_pp on pid 24578 2024-09-16T10:51:08.196 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_snapshots_pp 2024-09-16T10:51:08.197 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24578 2024-09-16T10:51:08.197 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_snapshots_pp on pid 24578' 2024-09-16T10:51:08.197 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24578 2024-09-16T10:51:08.197 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.197 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.198 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T10:51:08.198 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_snapshots 2024-09-16T10:51:08.198 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T10:51:08.199 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_stat 2024-09-16T10:51:08.200 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_stat' 2024-09-16T10:51:08.200 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T10:51:08.200 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_stat 2024-09-16T10:51:08.200 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.203 INFO:tasks.workunit.client.0.smithi143.stdout:test api_stat on pid 24593 2024-09-16T10:51:08.203 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_stat 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24593 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_stat on pid 24593' 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24593 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_stat_pp 2024-09-16T10:51:08.204 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_stat_pp' 2024-09-16T10:51:08.205 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.205 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_stat_pp 2024-09-16T10:51:08.205 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.207 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.207 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.207 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.207 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_stat_pp 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24612 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_stat_pp on pid 24612' 2024-09-16T10:51:08.208 INFO:tasks.workunit.client.0.smithi143.stdout:test api_stat_pp on pid 24612 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24612 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_watch_notify 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_watch_notify' 2024-09-16T10:51:08.211 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_stat 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_watch_notify 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.212 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T10:51:08.213 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T10:51:08.213 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T10:51:08.218 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_watch_notify 2024-09-16T10:51:08.218 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24626 2024-09-16T10:51:08.218 INFO:tasks.workunit.client.0.smithi143.stdout:test api_watch_notify on pid 24626 2024-09-16T10:51:08.219 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_watch_notify on pid 24626' 2024-09-16T10:51:08.219 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24626 2024-09-16T10:51:08.219 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.219 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.221 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_stat_pp 2024-09-16T10:51:08.222 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T10:51:08.222 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_watch_notify_pp' 2024-09-16T10:51:08.225 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T10:51:08.229 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_watch_notify_pp 2024-09-16T10:51:08.229 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T10:51:08.231 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T10:51:08.235 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_watch_notify_pp 2024-09-16T10:51:08.235 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24651 2024-09-16T10:51:08.235 INFO:tasks.workunit.client.0.smithi143.stdout:test api_watch_notify_pp on pid 24651 2024-09-16T10:51:08.236 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_watch_notify_pp on pid 24651' 2024-09-16T10:51:08.236 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24651 2024-09-16T10:51:08.236 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.236 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.237 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_watch_notify 2024-09-16T10:51:08.237 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T10:51:08.240 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.240 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.241 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.241 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.241 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cmd 2024-09-16T10:51:08.242 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cmd' 2024-09-16T10:51:08.242 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T10:51:08.242 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T10:51:08.242 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 10 monclient: get_monmap_and_config 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 10 monclient: build_initial_monmap 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 10 monclient: monmap: 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 0 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:08.243 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 0.000000 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 0.000000 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 1 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 1 Processor -- start 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 1 -- start start 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.236+0000 7f852598d980 10 monclient: init 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient: _reopen_session rank -1 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:51:08.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 0x5593714e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): picked mon.noname-b con 0x5593714cc8d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 0x5593714d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): picked mon.noname-c con 0x5593715856a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 0x5593714d50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): picked mon.noname-a con 0x559371585a80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593714662f0 con 0x559371585a80 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593714228e0 con 0x5593714cc8d0 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x559371422580 con 0x5593715856a0 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): _renew_subs 2024-09-16T10:51:08.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f852598d980 10 monclient(hunting): authenticate will time out at 928.660777s 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f85234c5700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 msgr2=0x5593714e27b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f85234c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 0x5593714e27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 0x5593714d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 0x5593714d50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 0x5593714d4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.143:50338/0 (socket says 172.21.15.143:50338) 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 1 -- 172.21.15.143:0/827613403 learned_addr learned my addr 172.21.15.143:0/827613403 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): get_auth_request con 0x5593715856a0 auth_method 0 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): get_auth_request con 0x559371585a80 auth_method 0 2024-09-16T10:51:08.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8522cc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient(hunting): handle_auth_done global_id 4539 payload 274 2024-09-16T10:51:08.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 -- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 msgr2=0x5593714d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:08.249 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-16T10:51:08.249 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T10:51:08.249 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T10:51:08.249 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cmd 2024-09-16T10:51:08.251 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.254 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cmd 2024-09-16T10:51:08.254 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 0x5593714d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0test api_cmd on pid 24716 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24716 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cmd on pid 24716' 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24716 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.255 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cmd 2024-09-16T10:51:08.256 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T10:51:08.257 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cmd_pp 2024-09-16T10:51:08.257 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T10:51:08.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cmd_pp' 2024-09-16T10:51:08.259 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cmd_pp 2024-09-16T10:51:08.259 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cmd_pp 2024-09-16T10:51:08.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24734 2024-09-16T10:51:08.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cmd_pp on pid 24734' 2024-09-16T10:51:08.264 INFO:tasks.workunit.client.0.smithi143.stdout:test api_cmd_pp on pid 24734 2024-09-16T10:51:08.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24734 2024-09-16T10:51:08.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T10:51:08.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.267 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_service 2024-09-16T10:51:08.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-16T10:51:08.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_service' 2024-09-16T10:51:08.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T10:51:08.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T10:51:08.269 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.269 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_service 2024-09-16T10:51:08.271 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_service 2024-09-16T10:51:08.271 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24748 2024-09-16T10:51:08.272 INFO:tasks.workunit.client.0.smithi143.stdout:test api_service on pid 24748 2024-09-16T10:51:08.274 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_service on pid 24748' 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24748 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_service_pp 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_service_pp' 2024-09-16T10:51:08.275 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_service_pp 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.276 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.278 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_service_pp 2024-09-16T10:51:08.278 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24763 2024-09-16T10:51:08.278 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_service_pp on pid 24763' 2024-09-16T10:51:08.278 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T10:51:08.279 INFO:tasks.workunit.client.0.smithi143.stdout:test api_service_pp on pid 24763 2024-09-16T10:51:08.281 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24763 2024-09-16T10:51:08.281 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_service 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.282 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_c_write_operations 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_c_write_operations' 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_service_pp 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_service: /' 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_c_write_operations 2024-09-16T10:51:08.283 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.284 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_c_write_operations 2024-09-16T10:51:08.284 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24774 2024-09-16T10:51:08.284 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_c_write_operations on pid 24774' 2024-09-16T10:51:08.285 INFO:tasks.workunit.client.0.smithi143.stdout:test api_c_write_operations on pid 24774 2024-09-16T10:51:08.285 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24774 2024-09-16T10:51:08.285 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.286 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.286 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T10:51:08.286 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_c_read_operations 2024-09-16T10:51:08.286 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_c_read_operations' 2024-09-16T10:51:08.286 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.287 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.287 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.287 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.287 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.287 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_c_read_operations 2024-09-16T10:51:08.288 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-16T10:51:08.288 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T10:51:08.291 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T10:51:08.291 INFO:tasks.workunit.client.0.smithi143.stdout:test api_c_read_operations on pid 24795 2024-09-16T10:51:08.291 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_c_read_operations 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24795 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_c_read_operations on pid 24795' 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24795 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T10:51:08.292 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s api_cls_remote_reads 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' api_cls_remote_reads' 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.293 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo api_cls_remote_reads 2024-09-16T10:51:08.294 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T10:51:08.294 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.294 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-16T10:51:08.296 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T10:51:08.296 INFO:tasks.workunit.client.0.smithi143.stdout:test api_cls_remote_reads on pid 24809 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=api_cls_remote_reads 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24809 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test api_cls_remote_reads on pid 24809' 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24809 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.297 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-16T10:51:08.298 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s list_parallel 2024-09-16T10:51:08.298 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' list_parallel' 2024-09-16T10:51:08.298 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.298 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.298 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.299 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.299 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-16T10:51:08.299 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-16T10:51:08.299 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-16T10:51:08.300 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo list_parallel 2024-09-16T10:51:08.300 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.303 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=list_parallel 2024-09-16T10:51:08.303 INFO:tasks.workunit.client.0.smithi143.stdout:test list_parallel on pid 24822 2024-09-16T10:51:08.304 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24822 2024-09-16T10:51:08.304 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test list_parallel on pid 24822' 2024-09-16T10:51:08.304 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24822 2024-09-16T10:51:08.304 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.304 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.305 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T10:51:08.305 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s open_pools_parallel 2024-09-16T10:51:08.305 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' open_pools_parallel' 2024-09-16T10:51:08.306 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.306 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.306 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.306 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.313 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T10:51:08.313 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_list_parallel 2024-09-16T10:51:08.314 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T10:51:08.314 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo open_pools_parallel 2024-09-16T10:51:08.314 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.321 INFO:tasks.workunit.client.0.smithi143.stdout:test open_pools_parallel on pid 24855 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=open_pools_parallel 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24855 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test open_pools_parallel on pid 24855' 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24855 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.322 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 -- 172.21.15.143:0/827613403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 msgr2=0x5593714e27b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 0x5593714e27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient: _finish_hunting 0 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 monclient: found mon.noname-a 2024-09-16T10:51:08.325 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 -- 172.21.15.143:0/827613403 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c00a1c0 con 0x559371585a80 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.240+0000 7f8523cc6700 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 0x5593714d50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x5593714cc1e0 tx=0x7f851c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3953417c2cd22ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 1 -- 172.21.15.143:0/827613403 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c021410 con 0x559371585a80 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:08.326 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 10 monclient: dump: 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 1 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 18 (reef) 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 3 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 10 monclient: _finish_auth 0 2024-09-16T10:51:08.327 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 10 monclient: _check_auth_tickets 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f8521cc2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f852598d980 5 monclient: authenticate success, global_id 4539 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.276+0000 7f852598d980 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 1 -- 172.21.15.143:0/827613403 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851c01c070 con 0x559371585a80 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 1 -- 172.21.15.143:0/827613403 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851c021790 con 0x559371585a80 2024-09-16T10:51:08.328 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f8521cc2700 10 monclient: dump: 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 1 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.329 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 18 (reef) 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 3 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 10 monclient: get_monmap_and_config success 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 4 set_mon_vals no callback set 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 10 monclient: shutdown 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 -- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 msgr2=0x5593714d50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:08.330 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 0x5593714d50d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x5593714cc1e0 tx=0x7f851c011000 comp rx=0 tx=0).stop 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 -- 172.21.15.143:0/827613403 shutdown_connections 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x559371585a80 0x5593714d50d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593715856a0 0x5593714d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 --2- 172.21.15.143:0/827613403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593714cc8d0 0x5593714e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 -- 172.21.15.143:0/827613403 >> 172.21.15.143:0/827613403 conn(0x5593714c8180 msgr2=0x5593714ca170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.320+0000 7f852598d980 1 -- 172.21.15.143:0/827613403 shutdown_connections 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 -- 172.21.15.143:0/827613403 wait complete. 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient: build_initial_monmap 2024-09-16T10:51:08.331 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient: monmap: 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 0 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 0.000000 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 0.000000 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 1 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 librados: starting msgr at 2024-09-16T10:51:08.333 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.333 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.333 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.333 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 librados: starting objecter 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 Processor -- start 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 -- start start 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 librados: setting wanted keys 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 librados: calling monclient init 2024-09-16T10:51:08.334 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient: init 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient: _reopen_session rank -1 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 0x5593714e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): picked mon.noname-c con 0x5593714cc8d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5593715856a0 0x5593715816a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): picked mon.noname-a con 0x5593715856a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559371585a80 0x559371581be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.335 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): picked mon.noname-b con 0x559371585a80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593713f88d0 con 0x5593715856a0 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593714cafe0 con 0x559371585a80 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593714e25f0 con 0x5593714cc8d0 2024-09-16T10:51:08.336 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): _renew_subs 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f852598d980 10 monclient(hunting): authenticate will time out at 928.745506s 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8523cc6700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559371585a80 msgr2=0x559371581be0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8523cc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559371585a80 0x559371581be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 0x5593714e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 0x5593714e20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.143:50436/0 (socket says 172.21.15.143:50436) 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 -- 172.21.15.143:0/1058947703 learned_addr learned my addr 172.21.15.143:0/1058947703 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): get_auth_request con 0x5593714cc8d0 auth_method 0 2024-09-16T10:51:08.337 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8522cc4700 1 --2- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5593715856a0 0x5593715816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8522cc4700 10 monclient(hunting): get_auth_request con 0x5593715856a0 auth_method 0 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8522cc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8522cc4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f8522cc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:08.338 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 -- 172.21.15.143:0/1058947703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559371585a80 msgr2=0x559371581be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 --2- 172.21.15.143:0/1058947703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559371585a80 0x559371581be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 -- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5593715856a0 msgr2=0x5593715816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 --2- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x5593715856a0 0x5593715816a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient: _finish_hunting 0 2024-09-16T10:51:08.339 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 monclient: found mon.noname-c 2024-09-16T10:51:08.340 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:08.340 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:08.341 INFO:tasks.workunit.client.0.smithi143.stderr:++ printf %25s delete_pools_parallel 2024-09-16T10:51:08.342 INFO:tasks.workunit.client.0.smithi143.stderr:+ r=' delete_pools_parallel' 2024-09-16T10:51:08.342 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T10:51:08.342 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T10:51:08.342 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55937142aca0 con 0x5593714cc8d0 2024-09-16T10:51:08.342 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.324+0000 7f85234c5700 1 --2- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 0x5593714e20b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8514008700 tx=0x7f8514008ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 client.?.objecter ms_handle_connect 0x5593714cc8d0 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85140067c0 con 0x5593714cc8d0 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: dump: 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 1 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 18 (reef) 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 3 2024-09-16T10:51:08.343 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: _finish_auth 0 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: _check_auth_tickets 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 5 monclient: authenticate success, global_id 4481 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 monclient: _renew_subs 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559371410b00 con 0x5593714cc8d0 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 monclient: _renew_subs 2024-09-16T10:51:08.344 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559371403f40 con 0x5593714cc8d0 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 1 librados: init done 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f852598d980 10 librados: wait_for_osdmap waiting 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8514006920 con 0x5593714cc8d0 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8520cc0700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.328+0000 7f8521cc2700 4 set_mon_vals no callback set 2024-09-16T10:51:08.345 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stderr:++ awk '{print $1}' 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stderr:++ echo delete_pools_parallel 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514006cd0 con 0x5593714cc8d0 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 monclient: dump: 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: epoch 1 2024-09-16T10:51:08.346 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: min_mon_release 18 (reef) 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: election_strategy: 3 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:08.347 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f8514012430 con 0x5593714cc8d0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.143:6812/1098205034 conn(0x7f850c0351d0 0x7f850c037690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3534+0+0 (secure 0 0 0) 0x7f8514046150 con 0x5593714cc8d0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_map(17..17 src has 1..17) v4 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map got epochs [17,17] > 0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map decoding full epoch 17 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8522cc4700 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.143:6812/1098205034 conn(0x7f850c0351d0 0x7f850c037690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8522cc4700 10 monclient: get_auth_request con 0x7f850c0351d0 auth_method 0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 20 client.4481.objecter dump_active .. 0 homeless 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f852598d980 10 librados: wait_for_osdmap done waiting 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f852598d980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:08.348 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55937145edb0 con 0x5593714cc8d0 2024-09-16T10:51:08.349 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8522cc4700 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.143:6812/1098205034 conn(0x7f850c0351d0 0x7f850c037690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8518000f60 tx=0x7f8518001860 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:08.349 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 client.4481.objecter ms_handle_connect 0x7f850c0351d0 2024-09-16T10:51:08.349 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8514011ed0 con 0x5593714cc8d0 2024-09-16T10:51:08.349 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.332+0000 7f8520cc0700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-16T10:51:08.349 INFO:tasks.workunit.client.0.smithi143.stdout: api_atest delete_pools_parallel on pid 24938 2024-09-16T10:51:08.350 INFO:tasks.workunit.client.0.smithi143.stderr:+ ff=delete_pools_parallel 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24938 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ echo 'test delete_pools_parallel on pid 24938' 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ pids[$f]=24938 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ ret=0 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24370 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24370 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ '[' -z '' ']' 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.351 INFO:tasks.workunit.client.0.smithi143.stderr:+ case $- in 2024-09-16T10:51:08.352 INFO:tasks.workunit.client.0.smithi143.stderr:+ return 2024-09-16T10:51:08.352 INFO:tasks.workunit.client.0.smithi143.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T10:51:08.352 INFO:tasks.workunit.client.0.smithi143.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-16T10:51:08.352 INFO:tasks.workunit.client.0.smithi143.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T10:51:09.745 INFO:tasks.workunit.client.0.smithi143.stdout:sio: 2024-09-16T10:51:08.332+0000 7f85214c1700 10 client.4481.objecter _get_latest_version latest 17, have it 2024-09-16T10:51:09.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.336+0000 7f852598d980 10 client.4481.objecter create_pool name=ceph_test_rados_api_asio 2024-09-16T10:51:09.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.336+0000 7f852598d980 10 client.4481.objecter pool_op_submit 1 2024-09-16T10:51:09.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.336+0000 7f852598d980 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:08.336+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55937146fd30 con 0x5593714cc8d0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v19) v1 ==== 43+0+0 (secure 0 0 0) 0x7f8514010330 con 0x5593714cc8d0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 client.4481.objecter ms_dispatch 0x5593714560c0 pool_op_reply(tid 1 (0) Success v19) v1 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 client.4481.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v19) v1 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 client.4481.objecter have request 1 at 0x5593714db1d0 Op: create 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 20 client.4481.objecter waiting for client to reach epoch 19 before calling back 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 monclient: _renew_subs 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=18}) v3 -- 0x7f850c03a670 con 0x5593714cc8d0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.736+0000 7f8520cc0700 10 client.4481.objecter done 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== mon.2 v2:172.21.15.5:3301/0 8 ==== osd_map(18..19 src has 1..19) v4 ==== 14547+0+0 (secure 0 0 0) 0x7f851403a030 con 0x5593714cc8d0 2024-09-16T10:51:09.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_map(18..19 src has 1..19) v4 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map got epochs [18,19] > 17 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map decoding incremental epoch 18 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map decoding incremental epoch 19 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f8520cc0700 20 client.4481.objecter dump_active .. 0 homeless 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 10 librados: set snap read head -> 1 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 10 librados: writefull oid=exist nspace= 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371582120 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 19 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:09.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 1 --2- 1 api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjects (195 ms) 2024-09-16T10:51:10.432 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (41 ms) 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo2 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo3 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo4 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo5 2024-09-16T10:51:10.433 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo6,foo7 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo7 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo6 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo4 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo5 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo7 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo6 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo1 2024-09-16T10:51:10.434 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo1 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo2 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo3 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (141 ms) 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 1 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 10 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 13 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 7 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 14 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 0 0 2024-09-16T10:51:10.435 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 15 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 11 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 5 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 8 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 6 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 3 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 4 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 12 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 9 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2 0 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (136 ms) 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T10:51:10.436 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: x cursor=MIN 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=1 cursor=6:02547ec2:::1:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=10 cursor=6:52ea6a34:::10:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=13 cursor=6:566253c9:::13:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=7 cursor=6:5c6b0b28:::7:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=14 cursor=6:62a1935d:::14:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=0 cursor=6:6cac518f:::0:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=15 cursor=6:863748b0:::15:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=11 cursor=6:89d3ae78:::11:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=5 cursor=6:b29083e3:::5:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=8 cursor=6:bd63b0f1:::8:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=6 cursor=6:c4fdafeb:::6:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=3 cursor=6:cfc208b3:::3:head 2024-09-16T10:51:10.437 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=4 cursor=6:d83876eb:::4:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=12 cursor=6:de5d7c5f:::12:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=9 cursor=6:e960b815:::9:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > oid=2 cursor=6:f905c69b:::2:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=1 cursor=6:02547ec2:::1:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:02547ec2:::1:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:02547ec2:::1:head -> 1 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=10 cursor=6:52ea6a34:::10:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:52ea6a34:::10:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:52ea6a34:::10:head -> 10 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=13 cursor=6:566253c9:::13:head 2024-09-16T10:51:10.438 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:566253c9:::13:head 2024-09-16T10:51:10.439 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-09-16T10:51:10.439 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=7 cursor=6:5c6b0b28:::7:head 2024-09-16T10:51:10.439 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:5c6b0b28:::7:head 2024-09-16T10:51:10.439 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:5c6b0b28:::7:head -> 7 2024-09-16T10:51:10.439 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=14 cursor=6:62a1935d:::14:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:62a1935d:::14:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:62a1935d:::14:head -> 14 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=0 cursor=6:6cac518f:::0:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=15 cursor=6:863748b0:::15:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:863748b0:::15:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=11 cursor=6:89d3ae78:::11:head 2024-09-16T10:51:10.594 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=5 cursor=6:b29083e3:::5:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=8 cursor=6:bd63b0f1:::8:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=6 cursor=6:c4fdafeb:::6:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:c4fdafeb:::6:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:c4fdafeb:::6:head -> 6 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=3 cursor=6:cfc208b3:::3:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:cfc208b3:::3:head 2024-09-16T10:51:10.595 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:cfc208b3:::3:head -> 3 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=4 cursor=6:d83876eb:::4:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=12 cursor=6:de5d7c5f:::12:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=9 cursor=6:e960b815:::9:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:e960b815:::9:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:e960b815:::9:head -> 9 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : oid=2 cursor=6:f905c69b:::2:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-09-16T10:51:10.596 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:f905c69b:::2:head -> 2 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:52ea6a34:::10:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:52ea6a34:::10:head expected=6:52ea6a34:::10:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:52ea6a34:::10:head -> 10 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=10 expected=10 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:02547ec2:::1:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:02547ec2:::1:head expected=6:02547ec2:::1:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:02547ec2:::1:head -> 1 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=1 expected=1 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:de5d7c5f:::12:head expected=6:de5d7c5f:::12:head 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=12 expected=12 2024-09-16T10:51:10.597 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:d83876eb:::4:head expected=6:d83876eb:::4:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=4 expected=4 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:cfc208b3:::3:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:cfc208b3:::3:head expected=6:cfc208b3:::3:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:cfc208b3:::3:head -> 3 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=3 expected=3 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:c4fdafeb:::6:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:c4fdafeb:::6:head expected=6:c4fdafeb:::6:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:c4fdafeb:::6:head -> 6 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=6 expected=6 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:863748b0:::15:head 2024-09-16T10:51:10.598 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:863748b0:::15:head expected=6:863748b0:::15:head 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=15 expected=15 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:6cac518f:::0:head expected=6:6cac518f:::0:head 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=0 expected=0 2024-09-16T10:51:10.599 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:62a1935d:::14:head 2024-09-16T10:51:10.868 INFO:tasks.workunit.client.0.smithi143.stdout: a72.21.15.143:0/1058947703 >> v2:172.21.15.5:6800/3675136489 conn(0x559371636c90 0x559371639080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:10.868 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 2 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 15 client.4481.objecter _send_op 2 to 19.1 on osd.1 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:2 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x5593714d5280 con 0x559371636c90 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.740+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.744+0000 7f8523cc6700 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.5:6800/3675136489 conn(0x559371636c90 0x559371639080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.744+0000 7f8523cc6700 10 monclient: get_auth_request con 0x559371636c90 auth_method 0 2024-09-16T10:51:10.869 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.752+0000 7f8523cc6700 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.5:6800/3675136489 conn(0x559371636c90 0x559371639080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:09.752+0000 7f8520cc0700 10 client.4481.objecter ms_handle_connect 0x559371636c90 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 1 ==== osd_map(20..20 src has 1..20) v4 ==== 5875+0+0 (crc 0 0 0) 0x7f851c02d810 con 0x559371636c90 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_map(20..20 src has 1..20) v4 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map got epochs [20,20] > 19 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 3 client.4481.objecter handle_osd_map decoding incremental epoch 20 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 10 client.4481.objecter checking op 2 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 20 client.4481.objecter dump_active .. 0 homeless 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.808+0000 7f8520cc0700 20 client.4481.objecter 2 19.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 2 ==== osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.870 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(2 exist [writefull 0~5] v20'1 uv1 ondisk = 0) v8 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 2 ondisk uv 1 in 19.1 attempt 0 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 2 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _finish_op 2 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 2 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 librados: Objecter returned from writefull r=0 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371582120 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.871 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 3 osd.1 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 3 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _send_op 3 to 19.1 on osd.1 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:3 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x55937157e380 con 0x559371636c90 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x55937157e730 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.872 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.873 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.873 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 4 osd.1 2024-09-16T10:51:10.873 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.873 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 4 2024-09-16T10:51:10.873 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _send_op 4 to 19.1 on osd.1 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:4 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x55937157ec20 con 0x559371636c90 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 3 ondisk uv 1 in 19.1 attempt 0 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 3 2024-09-16T10:51:10.874 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _finish_op 3 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 3 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 4 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 4 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _finish_op 4 2024-09-16T10:51:10.875 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 4 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x55937157e730 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 5 osd.1 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 5 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _send_op 5 to 19.1 on osd.1 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:5 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371582120 con 0x559371636c90 2024-09-16T10:51:10.876 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371641dd0 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 6 osd.1 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 6 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 15 client.4481.objecter _send_op 6 to 19.1 on osd.1 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:6 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5593716422c0 con 0x559371636c90 2024-09-16T10:51:10.877 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.864+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 5 ondisk uv 1 in 19.1 attempt 0 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter _finish_op 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 5 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.878 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 6 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 6 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter _finish_op 6 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 6 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (4 ms) 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371641dd0 2024-09-16T10:51:10.879 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 7 osd.1 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 7 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 15 client.4481.objecter _send_op 7 to 19.1 on osd.1 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:7 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x55937157e730 con 0x559371636c90 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.880 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371682d00 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 8 osd.1 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 8 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 15 client.4481.objecter _send_op 8 to 19.1 on osd.1 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:8 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x5593716831f0 con 0x559371636c90 2024-09-16T10:51:10.881 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.868+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 7 ondisk uv 1 in 19.1 attempt 0 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 7 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter _finish_op 7 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 7 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.882 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.883 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.883 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 8 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.907 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 8 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter _finish_op 8 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 8 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T10:51:10.908 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x55937157f050 write_seq 1 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371641dd0 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 9 osd.1 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 9 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 15 client.4481.objecter _send_op 9 to 19.1 on osd.1 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:9 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643680 con 0x559371636c90 2024-09-16T10:51:10.909 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.872+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 9 ==== osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(9 exist [write 0~5] v20'2 uv2 ondisk = 0) v8 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 9 ondisk uv 2 in 19.1 attempt 0 2024-09-16T10:51:10.910 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 9 2024-09-16T10:51:10.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 15 client.4481.objecter _finish_op 9 2024-09-16T10:51:10.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (32 ms) 2024-09-16T10:51:10.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 9 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f8523cc6700 20 librados: complete_aio_write 0x55937157f050 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x559371643a30 write_seq 2 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371641dd0 2024-09-16T10:51:10.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 10 osd.1 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 10 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 15 client.4481.objecter _send_op 10 to 19.1 on osd.1 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:10 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643680 con 0x559371636c90 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.904+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T10:51:10.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 10 ==== osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(10 exist [write 0~5] v20'3 uv3 ondisk = 0) v8 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 10 ondisk uv 3 in 19.1 attempt 0 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 10 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 15 client.4481.objecter _finish_op 10 2024-09-16T10:51:10.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (9 ms) 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 10 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f8523cc6700 20 librados: complete_aio_write 0x559371643a30 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x5593714d8840 write_seq 3 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371641dd0 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.925 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 11 osd.1 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 11 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 15 client.4481.objecter _send_op 11 to 19.1 on osd.1 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:11 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643680 con 0x559371636c90 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.912+0000 7f852598d980 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 11 ==== osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(11 exist [write 0~5] v20'4 uv4 ondisk = 0) v8 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.926 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 11 ondisk uv 4 in 19.1 attempt 0 2024-09-16T10:51:10.927 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.927 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 11 2024-09-16T10:51:10.927 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 15 client.4481.objecter _finish_op 11 2024-09-16T10:51:10.927 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.927 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 11 2024-09-16T10:51:10.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f8523cc6700 20 librados: complete_aio_write 0x5593714d8840 2024-09-16T10:51:10.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (9 ms) 2024-09-16T10:51:10.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T10:51:10.928 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371642670 2024-09-16T10:51:10.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.929 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 12 osd.1 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 12 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 15 client.4481.objecter _send_op 12 to 19.1 on osd.1 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:12 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371642b60 con 0x559371636c90 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371643020 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.930 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 13 osd.1 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 13 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 15 client.4481.objecter _send_op 13 to 19.1 on osd.1 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:13 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643510 con 0x559371636c90 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.920+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 12 ondisk uv 4 in 19.1 attempt 0 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.931 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 12 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 12 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 12 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 13 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 13 2024-09-16T10:51:10.932 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 13 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 13 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371643020 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 14 osd.1 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.933 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 14 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _send_op 14 to 19.1 on osd.1 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:14 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643510 con 0x559371636c90 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371642670 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 15 osd.1 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.934 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 15 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _send_op 15 to 19.1 on osd.1 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:15 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371642b60 con 0x559371636c90 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 14 ondisk uv 4 in 19.1 attempt 0 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 14 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 14 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.935 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 14 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 15 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 15 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 15 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 15 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.936 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x5593716659f0 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 16 osd.1 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 16 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _send_op 16 to 19.1 on osd.1 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:16 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371643020 con 0x559371636c90 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.937 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371666170 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 17 osd.1 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 17 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _send_op 17 to 19.1 on osd.1 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:17 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371666660 con 0x559371636c90 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.938 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 5 client.4481.objecter 2 in flight 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 16 ondisk uv 4 in 19.1 attempt 0 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 5 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 16 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 16 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 5 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 16 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-16T10:51:10.939 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 17 ondisk uv 0 in 19.1 attempt 0 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 17 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _finish_op 17 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.940 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 17 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (2 ms) 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x5593716437f0 write_seq 4 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371642670 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.941 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 18 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 15 client.4481.objecter _send_op 18 to 19.1 on osd.1 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:18 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371642b60 con 0x559371636c90 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.924+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 18 ==== osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(18 exist [writefull 0~5] v20'5 uv5 ondisk = 0) v8 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 18 ondisk uv 5 in 19.1 attempt 0 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 18 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 15 client.4481.objecter _finish_op 18 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.942 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 18 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f8523cc6700 20 librados: complete_aio_write 0x5593716437f0 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x5593716436a0 write_seq 5 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x559371642670 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.943 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 19 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 15 client.4481.objecter _send_op 19 to 19.1 on osd.1 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:19 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371642b60 con 0x559371636c90 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.928+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 19 ==== osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(19 exist [writefull 0~5] v20'6 uv6 ondisk = 0) v8 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 19 ondisk uv 6 in 19.1 attempt 0 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 19 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 15 client.4481.objecter _finish_op 19 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.944 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 19 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f8523cc6700 20 librados: complete_aio_write 0x5593716436a0 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6 ms) 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 librados: queue_aio_write 0x559371422370 completion 0x559371643020 write_seq 6 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 10 client.4481.objecter _op_submit op 0x5593716659f0 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 client.4481.objecter _calc_target epoch 20 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T10:51:10.945 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 client.4481.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 10 client.4481.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [1,5] primary 1 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 client.4481.objecter _get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 10 client.4481.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 client.4481.objecter get_session s=0x5593714da070 osd=1 3 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 15 client.4481.objecter _session_op_assign 1 20 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 15 client.4481.objecter _send_op 20 to 19.1 on osd.1 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 --> v2:172.21.15.5:6800/3675136489 -- osd_op(unknown.0.0:20 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x559371666170 con 0x559371636c90 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.932+0000 7f852598d980 5 client.4481.objecter 1 in flight 2024-09-16T10:51:10.946 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 <== osd.1 v2:172.21.15.5:6800/3675136489 20 ==== osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f851c027020 con 0x559371636c90 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 10 client.4481.objecter ms_dispatch 0x5593714560c0 osd_op_reply(20 exist [writefull 0~5] v20'7 uv7 ondisk = 0) v8 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 10 client.4481.objecter in handle_osd_op_reply 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 7 client.4481.objecter handle_osd_op_reply 20 ondisk uv 7 in 19.1 attempt 0 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 10 client.4481.objecter op 0 rval 0 len 0 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 15 client.4481.objecter handle_osd_op_reply completed tid 20 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 15 client.4481.objecter _finish_op 20 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 20 client.4481.objecter put_session s=0x5593714da070 osd=1 4 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 15 client.4481.objecter _session_op_remove 1 20 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 5 client.4481.objecter 0 in flight 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 20 librados: complete_aio_write 0x559371643020 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 10 librados: watch_flush enter 2024-09-16T10:51:10.947 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 10 librados: watch_flush exit 2024-09-16T10:51:10.948 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 10 client.4481.objecter close_session for osd.1 2024-09-16T10:51:10.948 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 >> v2:172.21.15.5:6800/3675136489 conn(0x559371636c90 msgr2=0x559371639080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:10.948 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.5:6800/3675136489 conn(0x559371636c90 0x559371639080 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:10.948 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 20 client.4481.objecter put_session s=0x5593714da070 osd=1 1 2024-09-16T10:51:10.948 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 20 client.4481.objecter shutdown clearing up homeless session... 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 10 client.4481.objecter successfully canceled tick 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 >> v2:172.21.15.143:6812/1098205034 conn(0x7f850c0351d0 msgr2=0x7f850c037690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 --2- 172.21.15.143:0/1058947703 >> v2:172.21.15.143:6812/1098205034 conn(0x7f850c0351d0 0x7f850c037690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8518000f60 tx=0x7f8518001860 comp rx=0 tx=0).stop 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 10 monclient: shutdown 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 msgr2=0x5593714e20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 --2- 172.21.15.143:0/1058947703 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x5593714cc8d0 0x5593714e20b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8514008700 tx=0x7f8514008ac0 comp rx=0 tx=0).stop 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f8523cc6700 1 -- 172.21.15.143:0/1058947703 reap_dead start 2024-09-16T10:51:10.956 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 shutdown_connections 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 >> 172.21.15.143:0/1058947703 conn(0x5593714c8180 msgr2=0x5593714e3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 shutdown_connections 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 -- 172.21.15.143:0/1058947703 wait complete. 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.940+0000 7f852598d980 1 librados: shutdown 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] 12 tests from AsioRados (74 ms total) 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2704 ms total) 2024-09-16T10:51:10.957 INFO:tasks.workunit.client.0.smithi143.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (3409 ms) 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (3409 ms total) 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (3409 ms total) 2024-09-16T10:51:11.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (49 ms) 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (49 ms) 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (3371 ms) 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3469 ms total) 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T10:51:11.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3469 ms total) 2024-09-16T10:51:11.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T10:51:11.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (133 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (12 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (15 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (27 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (8 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7 ms) 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T10:51:11.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (4 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (7 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (13 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (26 ms) 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T10:51:11.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (28 ms) 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (14 ms) 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (314 ms total) 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3446 ms total) 2024-09-16T10:51:11.764 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T10:51:11.841 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T10:51:11.841 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T10:51:11.841 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T10:51:11.841 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (114 ms) 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (24 ms) 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3343 ms) 2024-09-16T10:51:11.842 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.110664+0000 mon.a [WRN] Health check failed: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.741537+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi143-24525-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.783930+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24719-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.784156+0000 mon.a [INF] from='client.? 172.21.15.143:0/3515918001' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi143-24805-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.784341+0000 mon.a [INF] from='client.? 172.21.15.143:0/3064542899' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24790-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.784465+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24742-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.784720+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24812-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.864865+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi143-24622-7"}]: dispatch 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.866314+0000 mon.c [INF] from='client.? 172.21.15.143:0/2018987372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.866892+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.871179+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi143-24622-7"}]: dispatch 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.876438+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-24622-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:11.843 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.877519+0000 mon.c [INF] from='client.? 172.21.15.143:0/2018987372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.878475+0000 list_parallel: process_1_[24933]: starting. 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: creating pool ceph_test_rados_list_parallel.smithi143-24828 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: created object 0... 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: created object 25... 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: created object 49... 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: finishing. 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_1_[24933]: shutting down. 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: starting. 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: listing objects. 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: listed object 0... 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: listed object 25... 2024-09-16T10:51:11.962 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: saw 50 objects 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_2_[24937]: shutting down. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: starting. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: creating pool ceph_test_rados_list_parallel.smithi143-24828 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: created object 0... 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: created object 25... 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: created object 49... 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: finishing. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_3_[25340]: shutting down. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: starting. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: listing objects. 2024-09-16T10:51:11.963 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: listed object 0... 2024-09-16T10:51:11.964 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: listed object 25... 2024-09-16T10:51:11.964 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: saw 45 objects 2024-09-16T10:51:11.964 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_4_[25341]: shutting down. 2024-09-16T10:51:11.964 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.964 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: starting. 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: removed 25 objects... 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: removed half of the objects 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: removed 50 objects... 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: removed 50 objects 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_5_[25342]: shutting down. 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: starting. 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: creating pool ceph_test_rados_list_parallel.smithi143-24828 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: created object 0... 2024-09-16T10:51:11.965 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: created object 25... 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: created object 49... 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: finishing. 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_6_[25391]: shutting down. 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: starting. 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: listing objects. 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: listed object 0... 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: listed object 25... 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: listed object 50... 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: saw 56 objects 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_7_[25392]: shutting down. 2024-09-16T10:51:11.966 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: starting. 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: added 25 objects... 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: added half of the objects 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: added 50 objects... 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: added 50 objects 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_8_[25393]: shutting down. 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:11.967 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.359 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.359 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: starting. 2024-09-16T10:51:12.359 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: creating pool ceph_test_rados_list_parallel.smithi143-24828 2024-09-16T10:51:12.359 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: created object 0... 2024-09-16T10:51:12.360 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: created object 25... 2024-09-16T10:51:12.360 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: created object 49... 2024-09-16T10:51:12.360 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: finishing. 2024-09-16T10:51:12.360 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_9_[25442]: shutting down. 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: starting. 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listing objects. 2024-09-16T10:51:12.361 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listed object 0... 2024-09-16T10:51:12.362 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listed object 25... 2024-09-16T10:51:12.362 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listed object 50... 2024-09-16T10:51:12.362 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listed object 75... 2024-09-16T10:51:12.362 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: listed object 100... 2024-09-16T10:51:12.362 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: saw 102 objects 2024-09-16T10:51:12.363 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_10_[25443]: shutting down. 2024-09-16T10:51:12.363 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.363 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.363 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.363 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: starting. 2024-09-16T10:51:12.364 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: removed 25 objects... 2024-09-16T10:51:12.364 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: removed half of the objects 2024-09-16T10:51:12.364 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: removed 50 objects... 2024-09-16T10:51:12.364 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: removed 50 objects 2024-09-16T10:51:12.364 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_13_[25446]: shutting down. 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: starting. 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: added 25 objects... 2024-09-16T10:51:12.365 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: added half of the objects 2024-09-16T10:51:12.366 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: added 50 objects... 2024-09-16T10:51:12.366 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: added 50 objects 2024-09-16T10:51:12.366 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_12_[25445]: shutting down. 2024-09-16T10:51:12.366 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.366 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.367 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.367 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: starting. 2024-09-16T10:51:12.367 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: added 25 objects... 2024-09-16T10:51:12.367 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: added half of the objects 2024-09-16T10:51:12.367 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: added 50 objects... 2024-09-16T10:51:12.368 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: added 50 objects 2024-09-16T10:51:12.368 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_11_[25444]: shutting down. 2024-09-16T10:51:12.368 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.368 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.368 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: starting. 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: creating pool ceph_test_rados_list_parallel.smithi143-24828 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: created object 0... 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: created object 25... 2024-09-16T10:51:12.369 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: created object 49... 2024-09-16T10:51:12.370 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: finishing. 2024-09-16T10:51:12.370 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_14_[25644]: shutting down. 2024-09-16T10:51:12.370 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.370 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.370 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.371 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.371 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: starting. 2024-09-16T10:51:12.371 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listing objects. 2024-09-16T10:51:12.749 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed objecmon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.880390+0000 mon.c [INF] from='client.? 172.21.15.143:0/2018987372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-24613-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.881461+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-24613-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.900042+0000 mon.c [INF] from='client.? 172.21.15.143:0/3698673173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.938342+0000 mon.a [INF] from='client.? 172.21.15.143:0/3361686215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi143-24399-22"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.939602+0000 mon.a [INF] from='client.? 172.21.15.143:0/3361686215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi143-24399-22"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:10.940513+0000 mon.a [INF] from='client.? 172.21.15.143:0/3361686215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-24399-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.016330+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.017365+0000 mon.c [INF] from='client.? 172.21.15.143:0/3698673173' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:12.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.017880+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:12.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.018628+0000 mon.c [INF] from='client.? 172.21.15.143:0/3698673173' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-24391-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:12.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.053251+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-24391-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T10:51:12.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.744327+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi143-24665-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi143-24665-1"}]': finished 2024-09-16T10:51:12.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.744499+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi143-24622-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:12.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.744602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi143-24613-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.744788+0000 mon.a [INF] from='client.? 1t 0... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed object 25... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed object 50... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed object 75... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed object 100... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: listed object 125... 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: saw 150 objects 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_15_[25645]: shutting down. 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.915 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: starting. 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: added 25 objects... 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: added half of the objects 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: added 50 objects... 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: added 50 objects 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: process_16_[25646]: shutting down. 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.916 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******************************* 2024-09-16T10:51:12.917 INFO:tasks.workunit.client.0.smithi143.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T10:51:14.901 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: starting. 2024-09-16T10:51:14.901 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: creating pool ceph_test_rados_open_pools_parallel.smithi143-24875 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: created object 0... 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: created object 25... 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: created object 49... 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: finishing. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_1_[24956]: shutting down. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[24957]: starting. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[24957]: rados_pool_create. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[24957]: rados_ioctx_create. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_2_[24957]: shutting down. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: starting. 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: creating pool ceph_test_rados_open_pools_parallel.smithi143-24875 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: created object 0... 2024-09-16T10:51:14.902 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: created object 25... 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: created object 49... 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: finishing. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_3_[25527]: shutting down. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[25528]: starting. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[25528]: rados_pool_create. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[25528]: rados_ioctx_create. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: process_4_[25528]: shutting down. 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******************************* 2024-09-16T10:51:14.903 INFO:tasks.workunit.client.0.smithi143.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: starting. 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24942 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: created object 0... 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: created object 25... 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: created object 49... 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: finishing. 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_1_[24969]: shutting down. 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[24973]: starting. 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[24973]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24942 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_2_[24973]: shutting down. 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-09-16T10:51:14.904 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: starting. 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24942 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: created object 0... 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: created object 25... 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: created object 49... 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: finishing. 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_3_[25529]: shutting down. 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: starting. 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: listing objects. 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: listed object 0... 2024-09-16T10:51:14.905 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: listed object 25... 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: saw 50 objects 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_5_[25531]: shutting down. 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[25530]: starting. 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[25530]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24942 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: process_4_[25530]: shutting down. 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******************************* 2024-09-16T10:51:14.906 INFO:tasks.workunit.client.0.smithi143.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T10:51:14.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-16T10:51:14.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-16T10:51:14.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T10:51:14.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-16T10:51:14.915 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: seed 24399 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (124 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (31 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (3 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7 ms) 2024-09-16T10:51:14.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (6 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (10 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (6 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T10:51:14.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (26 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (15 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (7 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (6 ms) 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T10:51:14.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (5 ms) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (283 ms total) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1080 ms) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (20 ms) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (17 ms) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (6 ms) 2024-09-16T10:51:14.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T10:51:14.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-09-16T10:51:15.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN 72.21.15.143:0/3361686215' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi143-24399-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:15.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.745052+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi143-24391-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:15.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.746377+0000 mon.a [INF] from='client.? 172.21.15.143:0/3361686215' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-24399-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-24399-22"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.747035+0000 mon.c [INF] from='client.? 172.21.15.143:0/2018987372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-24613-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.747964+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-24622-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-24622-7"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.748288+0000 mon.c [INF] from='client.? 172.21.15.143:0/3698673173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-24391-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.750321+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-24613-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-24613-7"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.750797+0000 mon.a [INF] from='client.? 172.21.15.143:0/1865230172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-24383-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.751092+0000 mon.a [INF] from='client.? 172.21.15.143:0/4205569556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24525-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.751412+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-24391-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-24391-16"}]: dispatch 2024-09-16T10:51:15.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.751855+0000 mon.a [INF] from='client.? 172.21.15.143:0/1665572610' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24551-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.752213+0000 mon.a [INF] from='client.? 172.21.15.143:0/613727738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi143-24373-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.841519+0000 client.admin [INF] onexx 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:11.841665+0000 mon.a [INF] from='client.? 172.21.15.143:0/3043128460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.756167+0000 mon.a [INF] from='client.? 172.21.15.143:0/3043128460' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.770415+0000 mon.a [INF] from='client.? 172.21.15.143:0/3361686215' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi143-24399-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi143-24399-22"}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.770644+0000 mon.a [INF] from='client.? 172.21.15.143:0/3624689403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi143-24622-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi143-24622-7"}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.770773+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi143-24613-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi143-24613-7"}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.771195+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi143-24391-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi143-24391-16"}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.774585+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24790-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.774774+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi143-24438-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.774884+0000 mon.a [INF] from='client.4866 172.21.15.143:0/3741231716' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi143-24421-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.779148+0000 mon.a [INF] from='client.4866 172.21.15.143:0/3741231716' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi143-24421-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi143-24421-10"}]: dispatch 2024-09-16T10:51:15.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.791180+0000 mon.a [INF] from='client.? 172.21.15.143:0/4205569556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24525-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.791420+0000 mon.a [INF] from='client.? 172.21.15.143:0/1865230172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-24383-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.797230+0000 mon.c [INF] from='client.? 172.21.15.143:0/2799611459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-24438-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-24438-10"}]: dispatch 2024-09-16T10:51:15.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.797592+0000 mon.c [INF] from='client.? 172.21.15.143:0/2336001406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-24665-1"}]: dispatch 2024-09-16T10:51:15.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.798959+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi143-24438-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi143-24438-10"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:13.799780+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-24665-1"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.235394+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.236796+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.238111+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.239573+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.241115+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-09-16T10:51:15.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.242455+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.243714+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.245070+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.246463+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.247845+0000 mon.c [INF] from='client.? 172.21.15.143:0/3163949471' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.821498+0000 mon.a [INF] from='client.? 172.21.15.143:0/4205569556' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24525-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.821693+0000 mon.a [INF] from='client.? 172.21.15.143:0/1865230172' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi143-24383-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.890131+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi143-24665-1"}]': finished 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.891594+0000 mon.c [INF] from='client.? 172.21.15.143:0/2336001406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-24665-1"}]: dispatch 2024-09-16T10:51:15.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.892118+0000 mon.c [INF] from='client.? 172.21.15.143:0/2039825037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-24373-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:15.790 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.895037+0000 mon.a [INF] from='client.? 172.21.15.143:0/2656425630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi143-24551-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.896055+0000 mon.a [INF] from='client.? 172.21.15.143:0/4205569556' ent api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5133 ms) 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.RegisterLate (60 ms) 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T10:51:17.737 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: cluster: 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: id: 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: health: HEALTH_WARN 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 4 pool(s) do not have an application enabled 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: services: 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mon: 3 daemons, quorum a,c (age 49s), out of quorum: b 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mgr: x(active, since 78s) 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: osd: 6 osds: 6 up (since 75s), 6 in (since 82s) 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: data: 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pools: 19 pools, 553 pgs 2024-09-16T10:51:17.738 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pgs: 57.866% pgs unknown 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 40.506% pgs not active 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 320 unknown 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 224 creating+peering 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 9 active+clean 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: cluster: 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: id: 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: health: HEALTH_WARN 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 5 pool(s) do not have an application enabled 2024-09-16T10:51:17.739 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: services: 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mon: 3 daemons, quorum a,c (age 51s), out of quorum: b 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mgr: x(active, since 80s) 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: osd: 6 osds: 6 up (since 77s), 6 in (since 84s) 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: data: 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pools: 17 pools, 465 pgs 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pgs: 63.656% pgs unknown 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 34.409% pgs not active 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 296 unknown 2024-09-16T10:51:17.740 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 160 creating+peering 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 9 active+clean 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: cluster: 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: id: 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: health: HEALTH_WARN 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 9 pool(s) do not have an application enabled 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: services: 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mon: 3 daemons, quorum a,c (age 53s), out of quorum: b 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: mgr: x(active, since 82s) 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: osd: 6 osds: 6 up (since 79s), 6 in (since 86s) 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T10:51:17.741 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: data: 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pools: 21 pools, 521 pgs 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: objects: 63 objects, 145 MiB 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: usage: 183 MiB used, 540 GiB / 540 GiB avail 2024-09-16T10:51:17.742 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: pgs: 18.426% pgs unknown 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 24.568% api_stat: Running main() from gmock_main.cc 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.Stat (135 ms) 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (14 ms) 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.StatNS (11 ms) 2024-09-16T10:51:17.826 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (5 ms) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 5 tests from LibRadosStat (166 ms total) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1056 ms) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (31 ms) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-09-16T10:51:17.827 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1092 ms total) 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9594 ms total) 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T10:51:17.828 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: seed 24622 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (123 ms) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (5 ms) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5 ms) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (25 ms) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (158 ms total) 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: 2024-09-16T10:51:17.829 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1079 ms) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (21 ms) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (17 ms) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (20 ms) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1137 ms total) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9580 ms total) 2024-09-16T10:51:17.830 INFO:tasks.workunit.client.0.smithi143.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (15 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (9 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (7 ms) 2024-09-16T10:51:17.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (7 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (16 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (15 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (12 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (7 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (9 ms) 2024-09-16T10:51:17.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1241 ms total) 2024-09-16T10:51:17.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: 2024-09-16T10:51:17.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T10:51:17.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (9681 ms total) 2024-09-16T10:51:17.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: Running main() from gmock_main.cc 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (118 ms) 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T10:51:17.834 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: no timeout :/ 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (37 ms) 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (19 ms) 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7 ms) 2024-09-16T10:51:17.835 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (7 ms) 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (7 ms) 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (5 ms) 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T10:51:17.836 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.RmXattr (16 ms) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIo.XattrIter (14 ms) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 14 tests from LibRadosIo (249 ms total) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1277 ms) 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T10:51:17.837 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (11 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (7 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (12 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (7 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (7 ms) 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T10:51:17.838 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (12 ms) 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (30 ms) 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (11 ms) 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1383 ms total) 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [----------] Global test environment tear-down 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9691 ms total) 2024-09-16T10:51:17.839 INFO:tasks.workunit.client.0.smithi143.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout:ity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi143-24525-4", "tierpool": "test-rados-api-smithi143-24525-4-cache"}]: dispatch 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.899063+0000 mon.a [INF] from='client.? 172.21.15.143:0/1865230172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi143-24383-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.902040+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi143-24665-1"}]: dispatch 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:14.903008+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi143-24373-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:15.757275+0000 client.admin [INF] threexx 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: got: 2024-09-16T10:51:15.757803+0000 mon.a [INF] from='client.? 172.21.15.143:0/3043128460' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7046 ms) 2024-09-16T10:51:18.801 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10527 ms total) 2024-09-16T10:51:18.802 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: 2024-09-16T10:51:18.802 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T10:51:18.802 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10527 ms total) 2024-09-16T10:51:18.802 INFO:tasks.workunit.client.0.smithi143.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: seed 24438 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (116 ms) 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (18 ms) 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1066 ms) 2024-09-16T10:51:21.671 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1008 ms) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (8 ms) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6 ms) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (7 ms) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5 ms) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2234 ms total) 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T10:51:21.672 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1062 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (12 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1011 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1013 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (8 ms) 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T10:51:21.673 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (11 ms) 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (8 ms) 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3134 ms total) 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13493 ms total) 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T10:51:21.674 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (131 ms) 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockShared (27 ms) 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1065 ms) 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1005 ms) 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T10:51:21.675 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (7 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.Unlock (11 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (8 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (10 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLock (2265 ms total) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1064 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (5 ms) 2024-09-16T10:51:21.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1007 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1010 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (213 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (13 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (14 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (11 ms) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3337 ms total) 2024-09-16T10:51:21.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: 2024-09-16T10:51:21.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T10:51:21.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13508 ms total) 2024-09-16T10:51:21.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1221 ms) 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (10 ms) 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1231 ms total) 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (59 ms) 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3840 ms) 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (10 ms) 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (7 ms) 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775700448 notify_id 133143986178 notifier_gid 4832 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (9 ms) 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775697296 notify_id 133143986176 notifier_gid 4832 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (11 ms) 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775697296 notify_id 133143986180 notifier_gid 4832 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (13 ms) 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775697296 notify_id 133143986179 notifier_gid 4832 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (12 ms) 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775717616 notify_id 133143986177 notifier_gid 4832 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775717616 notify_id 133143986178 notifier_gid 4832 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (12 ms) 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775714464 notify_id 133143986180 notifier_gid 4832 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (15 ms) 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: trying... 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775714464 notify_id 133143986179 noti api_watch_notify: Running main() from gmock_main.cc 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T10:51:28.665 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (154 ms) 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94591869020048 err -107 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1019 ms) 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94591869811712 err -107 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1008 ms) 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313216 cookie 94591869825920 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T10:51:28.666 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313216 cookie 94591869825920 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (9 ms) 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313217 cookie 94591869825920 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (8 ms) 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313217 cookie 94591869825920 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313217 cookie 94591869859040 2024-09-16T10:51:28.667 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (14 ms) 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 90194313217 cookie 94591869825920 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 94489280514 cookie 94591869825920 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3009 ms) 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94591869825920 err -107 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_cb from 4590 notify_id 124554051587 cookie 94591869845376 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5018 ms) 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T10:51:28.668 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94591869825920 err -107 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1012 ms) 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11260 ms total) 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1054 ms) 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1054 ms total) 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20404 ms total) 2024-09-16T10:51:28.669 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: 2024-09-16T10:51:29.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3445 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3143 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2926 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2841 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3034 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2981 ms) 2024-09-16T10:51:29.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (2998 ms) 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (21368 ms total) 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (21368 ms total) 2024-09-16T10:51:29.683 INFO:tasks.workunit.client.0.smithi143.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout:fier_gid 4832 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: timed out 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushed 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3009 ms) 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: trying... 2024-09-16T10:51:37.676 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775714464 notify_id 146028888068 notifier_gid 4832 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: timed out 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushed 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3011 ms) 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: List watches 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775714464 notify_id 158913789957 notifier_gid 4832 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 done 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: watch_check 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: unwatch2 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-09-16T10:51:37.677 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: done 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3012 ms) 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: List watches 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: handle_notify cookie 94347775714464 notify_id 171798691845 notifier_gid 4832 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: notify2 done 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: watch_check 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: unwatch2 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: flushing 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: done 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3021 ms) 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16041 ms total) 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: 2024-09-16T10:51:37.678 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T10:51:37.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (29404 ms total) 2024-09-16T10:51:37.679 INFO:tasks.workunit.client.0.smithi143.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T10:51:37.831 INFO:tasks.workunit.client.0.smithi143.stdout: pgs not active 2024-09-16T10:51:37.831 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 297 active+clean 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 128 creating+peering 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 96 unknown 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: io: 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: client: 12 MiB/s rd, 37 MiB/s wr, 254 op/s rd, 291 op/s wr 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4270 ms) 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ OK ] LibRadosService.Status (20050 ms) 2024-09-16T10:51:37.832 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] 4 tests from LibRadosService (29514 ms total) 2024-09-16T10:51:37.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: 2024-09-16T10:51:37.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T10:51:37.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29514 ms total) 2024-09-16T10:51:37.833 INFO:tasks.workunit.client.0.smithi143.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T10:51:38.959 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T10:51:38.959 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T10:51:38.959 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T10:51:38.959 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T10:51:38.959 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5130 ms) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (51 ms) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20057 ms) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: attempt 0 of 20 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5405 ms) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30643 ms total) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30643 ms total) 2024-09-16T10:51:38.960 INFO:tasks.workunit.client.0.smithi143.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T10:51:44.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T10:51:44.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:08.300+0000 7f75d0f8e980 -1 failed for service _ceph-mon._tcp 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:08.300+0000 7f75d0f8e980 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (99 ms) 2024-09-16T10:51:44.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5018 ms) 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5117 ms total) 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 0x55b8e1337050 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 0x55b8e0905350 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: started 2 aios 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: waiting 0x55b8e1337050 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: waiting 0x55b8e0905350 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: done. 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5336 ms) 2024-09-16T10:51:44.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5336 ms total) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.Exec (44 ms) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (12 ms) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4944 ms) 2024-09-16T10:51:44.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (18005 ms) 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] 8 tests from LibRadosMisc (23009 ms total) 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (36482 ms total) 2024-09-16T10:51:44.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T10:51:45.680 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2589 ms) 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (2988 ms) 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3029 ms) 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2842 ms) 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T10:51:45.681 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3017 ms) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5055 ms) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4951 ms) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4998 ms) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7996 ms) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] 9 tests from LibRadosPools (37466 ms total) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37466 ms total) 2024-09-16T10:51:45.682 INFO:tasks.workunit.client.0.smithi143.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T10:51:55.234 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:51:55.234 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T10:51:55.234 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:51:55.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:51:55.238 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:51:55.238 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:51:55.238 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:51:55.343 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T10:51:55.338+0000 7f66adc1dd40 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 20 -> 0 (sync_force? mon rebuild?) 2024-09-16T10:51:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 10 monclient: get_monmap_and_config 2024-09-16T10:51:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 10 monclient: build_initial_monmap 2024-09-16T10:51:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 10 monclient: monmap: 2024-09-16T10:51:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:55.362940+0000 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:55.362940+0000 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:55.364 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 1 Processor -- start 2024-09-16T10:51:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 1 -- start start 2024-09-16T10:51:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.360+0000 7f19bf286700 10 monclient: init 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f19b806efe0 0x7f19b8131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-a con 0x7f19b806efe0 addr v2:172.21.15.5:3300/0 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-b con 0x7f19b80a0ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19b812e9a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b8070e60 con 0x7f19b812e9a0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b8132990 con 0x7f19b806efe0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b8070cd0 con 0x7f19b80a0ab0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient(hunting): authenticate will time out at 973.941295s 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46546/0 (socket says 172.21.15.5:46546) 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 -- 172.21.15.5:0/829998444 learned_addr learned my addr 172.21.15.5:0/829998444 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): get_auth_request con 0x7f19b80a0ab0 auth_method 0 2024-09-16T10:51:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 1 --2- 172.21.15.5:0/829998444 >> v2:172.21.15.5:3300/0 conn(0x7f19b806efe0 0x7f19b8131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): get_auth_request con 0x7f19b806efe0 auth_method 0 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19be284700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient(hunting): handle_auth_done global_id 22130 payload 274 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 -- 172.21.15.5:0/829998444 >> v2:172.21.15.5:3300/0 conn(0x7f19b806efe0 msgr2=0x7f19b8131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 --2- 172.21.15.5:0/829998444 >> v2:172.21.15.5:3300/0 conn(0x7f19b806efe0 0x7f19b8131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 -- 172.21.15.5:0/829998444 >> v1:172.21.15.143:6789/0 conn(0x7f19b812e9a0 legacy=0x7f19b8138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient: _finish_hunting 0 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 monclient: found mon.noname-b 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 -- 172.21.15.5:0/829998444 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b806f3c0 con 0x7f19b80a0ab0 2024-09-16T10:51:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bda83700 1 --2- 172.21.15.5:0/829998444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 secure :-1 s=READY pgs=4485 cs=0 l=1 rev1=1 crypto rx=0x7f19a8006fd0 tx=0x7f19a800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aecd8a99b57136b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 1 -- 172.21.15.5:0/829998444 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a800dc20 con 0x7f19b80a0ab0 2024-09-16T10:51:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:51:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: dump: 2024-09-16T10:51:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: _finish_auth 0 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: _check_auth_tickets 2024-09-16T10:51:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 5 monclient: authenticate success, global_id 22130 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 1 -- 172.21.15.5:0/829998444 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a800dd80 con 0x7f19b80a0ab0 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 4 set_mon_vals no callback set 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 1 -- 172.21.15.5:0/829998444 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8004250 con 0x7f19b80a0ab0 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bd282700 10 monclient: dump: 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 10 monclient: shutdown 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- 172.21.15.5:0/829998444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 msgr2=0x7f19b812e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 --2- 172.21.15.5:0/829998444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 secure :-1 s=READY pgs=4485 cs=0 l=1 rev1=1 crypto rx=0x7f19a8006fd0 tx=0x7f19a800bea0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- 172.21.15.5:0/829998444 shutdown_connections 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 --2- 172.21.15.5:0/829998444 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b80a0ab0 0x7f19b812e460 unknown :-1 s=CLOSED pgs=4485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 --2- 172.21.15.5:0/829998444 >> v2:172.21.15.5:3300/0 conn(0x7f19b806efe0 0x7f19b8131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- 172.21.15.5:0/829998444 >> 172.21.15.5:0/829998444 conn(0x7f19b80a4880 msgr2=0x7f19b80a4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- 172.21.15.5:0/829998444 shutdown_connections 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.364+0000 7f19bf286700 1 -- 172.21.15.5:0/829998444 wait complete. 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: build_initial_monmap 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: monmap: 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:55.380 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 librados: starting msgr at 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 librados: starting objecter 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 Processor -- start 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- start start 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 librados: setting wanted keys 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 librados: calling monclient init 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: init 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-b con 0x7f19b806efe0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 0x7f19b81cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-a con 0x7f19b80a0ab0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): picked mon.noname-c con 0x7f19b812e9a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b81206e0 con 0x7f19b80a0ab0 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b80716d0 con 0x7f19b806efe0 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b8131070 con 0x7f19b812e9a0 2024-09-16T10:51:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient(hunting): authenticate will time out at 973.945882s 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46554/0 (socket says 172.21.15.5:46554) 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 -- 172.21.15.5:0/2504748296 learned_addr learned my addr 172.21.15.5:0/2504748296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 0x7f19b81cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): get_auth_request con 0x7f19b812e9a0 auth_method 0 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 10 monclient(hunting): get_auth_request con 0x7f19b806efe0 auth_method 0 2024-09-16T10:51:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): get_auth_request con 0x7f19b80a0ab0 auth_method 0 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 1 -- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 msgr2=0x7f19b81cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 1 -- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 msgr2=0x7f19b81cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19be284700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient(hunting): handle_auth_done global_id 22133 payload 824 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 -- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 msgr2=0x7f19b81cab80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 -- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 msgr2=0x7f19b81cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 0x7f19b81cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient: _finish_hunting 0 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 monclient: found mon.noname-c 2024-09-16T10:51:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 -- 172.21.15.5:0/2504748296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a800b820 con 0x7f19b812e9a0 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bda83700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 0x7f19b81cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bea85700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 secure :-1 s=READY pgs=4486 cs=0 l=1 rev1=1 crypto rx=0x7f19b000cc80 tx=0x7f19b000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 client.?.objecter ms_handle_connect 0x7f19b812e9a0 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0003c20 con 0x7f19b812e9a0 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: dump: 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: _finish_auth 0 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: _check_auth_tickets 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 5 monclient: authenticate success, global_id 22133 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _renew_subs 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b0003d80 con 0x7f19b812e9a0 2024-09-16T10:51:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b8120090 con 0x7f19b812e9a0 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000ab50 con 0x7f19b812e9a0 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 client.22133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19aeffd700 10 monclient: dump: 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _renew_subs 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b811faa0 con 0x7f19b812e9a0 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bf286700 1 librados: init done 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.368+0000 7f19bca81700 4 set_mon_vals no callback set 2024-09-16T10:51:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19bf286700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19bf286700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19bf286700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b806f850 con 0x7f19b812e9a0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f19b000acb0 con 0x7f19b812e9a0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 1 --2- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 0x7f199c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f19b0048460 con 0x7f19b812e9a0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 10 client.22133.objecter ms_dispatch 0x7f19b8000fd0 osd_map(64..64 src has 1..64) v4 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 3 client.22133.objecter handle_osd_map got epochs [64,64] > 0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 3 client.22133.objecter handle_osd_map decoding full epoch 64 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 20 client.22133.objecter dump_active .. 0 homeless 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19be284700 1 --2- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 0x7f199c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19be284700 10 monclient: get_auth_request con 0x7f199c036f60 auth_method 0 2024-09-16T10:51:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19aeffd700 10 client.22133.objecter ms_handle_connect 0x7f199c036f60 2024-09-16T10:51:55.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.372+0000 7f19be284700 1 --2- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 0x7f199c039420 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009590 tx=0x7f19b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:55.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.376+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f19b00198f0 con 0x7f19b812e9a0 2024-09-16T10:51:55.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.376+0000 7f19aeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:55.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.376+0000 7f19aeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:55.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19bf286700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:51:55.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19bf286700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:51:55.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19bf286700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:55.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f19b806a310 con 0x7f19b812e9a0 2024-09-16T10:51:55.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19aeffd700 1 -- 172.21.15.5:0/2504748296 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f19b0019300 con 0x7f19b812e9a0 2024-09-16T10:51:55.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19aeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:51:55.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.616+0000 7f19aeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:55.621 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 10 librados: watch_flush enter 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 10 librados: watch_flush exit 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 20 client.22133.objecter shutdown clearing up homeless session... 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 10 client.22133.objecter successfully canceled tick 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 msgr2=0x7f199c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 0x7f199c039420 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009590 tx=0x7f19b4008040 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 10 monclient: shutdown 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 msgr2=0x7f19b809e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 secure :-1 s=READY pgs=4486 cs=0 l=1 rev1=1 crypto rx=0x7f19b000cc80 tx=0x7f19b000cf90 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 shutdown_connections 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f19b812e9a0 0x7f19b809e1f0 unknown :-1 s=CLOSED pgs=4486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> v2:172.21.15.143:6812/1098205034 conn(0x7f199c036f60 0x7f199c039420 unknown :-1 s=CLOSED pgs=3652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19b80a0ab0 0x7f19b81cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 --2- 172.21.15.5:0/2504748296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b806efe0 0x7f19b81cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 >> 172.21.15.5:0/2504748296 conn(0x7f19b80a4880 msgr2=0x7f19b80a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 shutdown_connections 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 -- 172.21.15.5:0/2504748296 wait complete. 2024-09-16T10:51:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:55.620+0000 7f19bf286700 1 librados: shutdown 2024-09-16T10:51:58.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:51:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: get_monmap_and_config 2024-09-16T10:51:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: build_initial_monmap 2024-09-16T10:51:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: monmap: 2024-09-16T10:51:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:58.765323+0000 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:58.765323+0000 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:58.766 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 Processor -- start 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 -- start start 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: init 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a5c070ad0 0x7f9a5c070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-a con 0x7f9a5c070ad0 addr v2:172.21.15.5:3300/0 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-c con 0x7f9a5c09dc30 addr v2:172.21.15.143:3300/0 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a5c0713f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c09e0b0 con 0x7f9a5c0713f0 2024-09-16T10:51:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c09f150 con 0x7f9a5c070ad0 2024-09-16T10:51:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c09d050 con 0x7f9a5c09dc30 2024-09-16T10:51:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a6136c700 10 monclient(hunting): authenticate will time out at 977.342290s 2024-09-16T10:51:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a5c070ad0 0x7f9a5c070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:33160/0 (socket says 172.21.15.5:33160) 2024-09-16T10:51:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 -- 172.21.15.5:0/1164555471 learned_addr learned my addr 172.21.15.5:0/1164555471 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): get_auth_request con 0x7f9a5c09dc30 auth_method 0 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): get_auth_request con 0x7f9a5c070ad0 auth_method 0 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_done global_id 34108 payload 274 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 -- 172.21.15.5:0/1164555471 >> v2:172.21.15.5:3300/0 conn(0x7f9a5c070ad0 msgr2=0x7f9a5c070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 --2- 172.21.15.5:0/1164555471 >> v2:172.21.15.5:3300/0 conn(0x7f9a5c070ad0 0x7f9a5c070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 -- 172.21.15.5:0/1164555471 >> v1:172.21.15.5:6789/0 conn(0x7f9a5c0713f0 legacy=0x7f9a5c0a1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 monclient: found mon.noname-c 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 -- 172.21.15.5:0/1164555471 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c09f9f0 con 0x7f9a5c09dc30 2024-09-16T10:51:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.764+0000 7f9a5b7fe700 1 --2- 172.21.15.5:0/1164555471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c074b90 tx=0x7f9a4c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d7c45639d86328 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 1 -- 172.21.15.5:0/1164555471 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c00e070 con 0x7f9a5c09dc30 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: dump: 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:58.783 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: _finish_auth 0 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: _check_auth_tickets 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 5 monclient: authenticate success, global_id 34108 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 1 -- 172.21.15.5:0/1164555471 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c003a40 con 0x7f9a5c09dc30 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 4 set_mon_vals no callback set 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 1 -- 172.21.15.5:0/1164555471 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c003df0 con 0x7f9a5c09dc30 2024-09-16T10:51:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5affd700 10 monclient: dump: 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:58.785 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: shutdown 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- 172.21.15.5:0/1164555471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c09dc30 msgr2=0x7f9a5c0a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1164555471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c074b90 tx=0x7f9a4c00b330 comp rx=0 tx=0).stop 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- 172.21.15.5:0/1164555471 shutdown_connections 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1164555471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c09dc30 0x7f9a5c0a19f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1164555471 >> v2:172.21.15.5:3300/0 conn(0x7f9a5c070ad0 0x7f9a5c070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- 172.21.15.5:0/1164555471 >> 172.21.15.5:0/1164555471 conn(0x7f9a5c0a3200 msgr2=0x7f9a5c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- 172.21.15.5:0/1164555471 shutdown_connections 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- 172.21.15.5:0/1164555471 wait complete. 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: build_initial_monmap 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: monmap: 2024-09-16T10:51:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 librados: starting msgr at 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 librados: starting objecter 2024-09-16T10:51:58.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 Processor -- start 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- start start 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 librados: setting wanted keys 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 librados: calling monclient init 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: init 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-c con 0x7f9a5c070ad0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c0713f0 0x7f9a5c1c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-b con 0x7f9a5c0713f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:58.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a5c1c8130 0x7f9a5c1c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): picked mon.noname-a con 0x7f9a5c1c8130 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1a9240 con 0x7f9a5c1c8130 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0a5270 con 0x7f9a5c0713f0 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0a2270 con 0x7f9a5c070ad0 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a6136c700 10 monclient(hunting): authenticate will time out at 977.347905s 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.768+0000 7f9a5bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46558/0 (socket says 172.21.15.5:46558) 2024-09-16T10:51:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 -- 172.21.15.5:0/1266384893 learned_addr learned my addr 172.21.15.5:0/1266384893 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): get_auth_request con 0x7f9a5c070ad0 auth_method 0 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a5c1c8130 0x7f9a5c1c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): get_auth_request con 0x7f9a5c1c8130 auth_method 0 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c0713f0 0x7f9a5c1c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): get_auth_request con 0x7f9a5c0713f0 auth_method 0 2024-09-16T10:51:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a60b6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient(hunting): handle_auth_done global_id 22145 payload 824 2024-09-16T10:51:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 -- 172.21.15.5:0/1266384893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c0713f0 msgr2=0x7f9a5c1c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c0713f0 0x7f9a5c1c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 -- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a5c1c8130 msgr2=0x7f9a5c1c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a5c1c8130 0x7f9a5c1c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 monclient: found mon.noname-c 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 -- 172.21.15.5:0/1266384893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c009b30 con 0x7f9a5c070ad0 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5bfff700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 secure :-1 s=READY pgs=4488 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400ebe0 tx=0x7f9a4400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 client.?.objecter ms_handle_connect 0x7f9a5c070ad0 2024-09-16T10:51:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44009740 con 0x7f9a5c070ad0 2024-09-16T10:51:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: dump: 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: _finish_auth 0 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 5 monclient: authenticate success, global_id 22145 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _renew_subs 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:58.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c09d1d0 con 0x7f9a5c070ad0 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4400c850 con 0x7f9a5c070ad0 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4400cbe0 con 0x7f9a5c070ad0 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a58ff9700 10 monclient: dump: 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a5a7fc700 4 set_mon_vals no callback set 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 client.22145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _renew_subs 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c1a9ae0 con 0x7f9a5c070ad0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 1 librados: init done 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.772+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c070eb0 con 0x7f9a5c070ad0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9a4400cd40 con 0x7f9a5c070ad0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 1 --2- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 0x7f9a3c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7334+0+0 (secure 0 0 0) 0x7f9a44048e40 con 0x7f9a5c070ad0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 10 client.22145.objecter ms_dispatch 0x7f9a5c000fd0 osd_map(68..68 src has 1..68) v4 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 3 client.22145.objecter handle_osd_map got epochs [68,68] > 0 2024-09-16T10:51:58.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 3 client.22145.objecter handle_osd_map decoding full epoch 68 2024-09-16T10:51:58.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 20 client.22145.objecter dump_active .. 0 homeless 2024-09-16T10:51:58.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a5b7fe700 1 --2- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 0x7f9a3c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:58.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a5b7fe700 10 monclient: get_auth_request con 0x7f9a3c036f60 auth_method 0 2024-09-16T10:51:58.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a58ff9700 10 client.22145.objecter ms_handle_connect 0x7f9a3c036f60 2024-09-16T10:51:58.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.776+0000 7f9a5b7fe700 1 --2- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 0x7f9a3c039420 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c000f80 tx=0x7f9a4c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:58.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.780+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a44011ce0 con 0x7f9a5c070ad0 2024-09-16T10:51:58.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.780+0000 7f9a58ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:58.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:58.780+0000 7f9a58ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:59.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.020+0000 7f9a6136c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.020+0000 7f9a6136c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.020+0000 7f9a6136c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.020+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9a5c1a77a0 con 0x7f9a5c070ad0 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a58ff9700 1 -- 172.21.15.5:0/1266384893 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f9a44017020 con 0x7f9a5c070ad0 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a58ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:51:59.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a58ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:59.026 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:51:59.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 10 librados: watch_flush enter 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 10 librados: watch_flush exit 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 20 client.22145.objecter shutdown clearing up homeless session... 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 10 client.22145.objecter successfully canceled tick 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 msgr2=0x7f9a3c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 0x7f9a3c039420 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c000f80 tx=0x7f9a4c009000 comp rx=0 tx=0).stop 2024-09-16T10:51:59.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 10 monclient: shutdown 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 msgr2=0x7f9a5c1c4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 secure :-1 s=READY pgs=4488 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400ebe0 tx=0x7f9a4400eef0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 shutdown_connections 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9a3c036f60 0x7f9a3c039420 unknown :-1 s=CLOSED pgs=3657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a5c1c8130 0x7f9a5c1c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a5c0713f0 0x7f9a5c1c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 --2- 172.21.15.5:0/1266384893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9a5c070ad0 0x7f9a5c1c4bf0 unknown :-1 s=CLOSED pgs=4488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 >> 172.21.15.5:0/1266384893 conn(0x7f9a5c0a3200 msgr2=0x7f9a5c0a5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 shutdown_connections 2024-09-16T10:51:59.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 -- 172.21.15.5:0/1266384893 wait complete. 2024-09-16T10:51:59.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.024+0000 7f9a6136c700 1 librados: shutdown 2024-09-16T10:51:59.043 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:51:59.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:51:59.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 10 monclient: get_monmap_and_config 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 10 monclient: monmap: 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:59.163837+0000 2024-09-16T10:51:59.164 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:59.163837+0000 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:59.165 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 1 Processor -- start 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 1 -- start start 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.160+0000 7f93eb9eb700 10 monclient: init 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T10:51:59.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-c con 0x7f93e40a2eb0 addr v2:172.21.15.143:3300/0 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93e40a4280 addr v1:172.21.15.5:6789/0 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 0x7f93e413d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-b con 0x7f93e413aeb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e409fdc0 con 0x7f93e40a4280 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e40a1e50 con 0x7f93e40a2eb0 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e409fae0 con 0x7f93e413aeb0 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient(hunting): authenticate will time out at 977.740754s 2024-09-16T10:51:59.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e8f86700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f93e40a4280 0x7f93e40a37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51400/0 (socket says 172.21.15.5:51400) 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e8f86700 1 -- 172.21.15.5:0/1628636451 learned_addr learned my addr 172.21.15.5:0/1628636451 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1628636451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 0x7f93e413d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): get_auth_request con 0x7f93e413aeb0 auth_method 0 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 1 --2- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 10 monclient(hunting): get_auth_request con 0x7f93e40a2eb0 auth_method 0 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 1 -- 172.21.15.5:0/1628636451 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277488656 0 0) 0x7f93e409fdc0 con 0x7f93e40a4280 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient(hunting): _negotiate 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient(hunting): my global_id is 27927 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 1 -- 172.21.15.5:0/1628636451 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d00035d0 con 0x7f93e40a4280 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 1 -- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 msgr2=0x7f93e40a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 1 -- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 msgr2=0x7f93e40a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 1 --2- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9787700 1 --2- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient(hunting): handle_auth_done global_id 22148 payload 274 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 -- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 msgr2=0x7f93e40a3290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:51:59.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 -- 172.21.15.5:0/1628636451 >> v1:172.21.15.5:6789/0 conn(0x7f93e40a4280 legacy=0x7f93e40a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 monclient: found mon.noname-b 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 -- 172.21.15.5:0/1628636451 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e409efb0 con 0x7f93e413aeb0 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1628636451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 0x7f93e413d2a0 secure :-1 s=READY pgs=4489 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009d90 tx=0x7f93dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d5e5d0346e21a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 1 -- 172.21.15.5:0/1628636451 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00e070 con 0x7f93e413aeb0 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: dump: 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.179 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 5 monclient: authenticate success, global_id 22148 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:59.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 1 -- 172.21.15.5:0/1628636451 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc00bb20 con 0x7f93e413aeb0 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 1 -- 172.21.15.5:0/1628636451 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc012430 con 0x7f93e413aeb0 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93dbfff700 10 monclient: dump: 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 4 set_mon_vals no callback set 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 10 monclient: shutdown 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1628636451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 msgr2=0x7f93e413d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1628636451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 0x7f93e413d2a0 secure :-1 s=READY pgs=4489 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009d90 tx=0x7f93dc00b330 comp rx=0 tx=0).stop 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1628636451 shutdown_connections 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1628636451 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e413aeb0 0x7f93e413d2a0 unknown :-1 s=CLOSED pgs=4489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1628636451 >> v2:172.21.15.143:3300/0 conn(0x7f93e40a2eb0 0x7f93e40a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1628636451 >> 172.21.15.5:0/1628636451 conn(0x7f93e40a4fb0 msgr2=0x7f93e40a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1628636451 shutdown_connections 2024-09-16T10:51:59.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.164+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1628636451 wait complete. 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: monmap: 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:59.183 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 librados: starting msgr at 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 librados: starting objecter 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 Processor -- start 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- start start 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 librados: setting wanted keys 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 librados: calling monclient init 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: init 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:51:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-c con 0x7f93e40a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 0x7f93e413a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-a con 0x7f93e40a4280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93e413aeb0 0x7f93e413a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): picked mon.noname-b con 0x7f93e413aeb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e41cf000 con 0x7f93e40a4280 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e40718f0 con 0x7f93e413aeb0 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e409f6a0 con 0x7f93e40a2eb0 2024-09-16T10:51:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): authenticate will time out at 977.745136s 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 0x7f93e413a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 0x7f93e413a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38290/0 (socket says 172.21.15.5:38290) 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46580/0 (socket says 172.21.15.5:46580) 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 1 -- 172.21.15.5:0/1065114412 learned_addr learned my addr 172.21.15.5:0/1065114412 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): get_auth_request con 0x7f93e40a4280 auth_method 0 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): get_auth_request con 0x7f93e40a2eb0 auth_method 0 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93e413aeb0 0x7f93e413a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): get_auth_request con 0x7f93e413aeb0 auth_method 0 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9f88700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient(hunting): handle_auth_done global_id 22151 payload 824 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93e413aeb0 msgr2=0x7f93e413a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f93e413aeb0 0x7f93e413a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 msgr2=0x7f93e413a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 0x7f93e413a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 monclient: found mon.noname-c 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc009a70 con 0x7f93e40a2eb0 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e40a4280 0x7f93e413a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:51:59.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e9787700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 secure :-1 s=READY pgs=4490 cs=0 l=1 rev1=1 crypto rx=0x7f93d400e7e0 tx=0x7f93d400eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 client.?.objecter ms_handle_connect 0x7f93e40a2eb0 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d400c070 con 0x7f93e40a2eb0 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: dump: 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.189 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d4004020 con 0x7f93e40a2eb0 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 5 monclient: authenticate success, global_id 22151 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _renew_subs 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e40711a0 con 0x7f93e40a2eb0 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4004410 con 0x7f93e40a2eb0 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 monclient: dump: 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.191 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93db7fe700 4 set_mon_vals no callback set 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 client.22151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _renew_subs 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e4128a20 con 0x7f93e40a2eb0 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 librados: init done 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 0x7f93e402b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e4067b80 con 0x7f93e406fd70 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f93d40036c0 con 0x7f93e40a2eb0 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 0x7f93e402b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 --2- 172.21.15.5:0/1065114412 >> v2:172.21.15.143:6812/1098205034 conn(0x7f93c8036f60 0x7f93c8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient: get_auth_request con 0x7f93e406fd70 auth_method 0 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93e8f86700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7334+0+0 (secure 0 0 0) 0x7f93d4047e90 con 0x7f93e40a2eb0 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.168+0000 7f93d9ffb700 10 client.22151.objecter ms_dispatch 0x7f93e4000fd0 osd_map(68..68 src has 1..68) v4 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 3 client.22151.objecter handle_osd_map got epochs [68,68] > 0 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 3 client.22151.objecter handle_osd_map decoding full epoch 68 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 20 client.22151.objecter dump_active .. 0 homeless 2024-09-16T10:51:59.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1065114412 >> v2:172.21.15.143:6812/1098205034 conn(0x7f93c8036f60 0x7f93c8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e9f88700 10 monclient: get_auth_request con 0x7f93c8036f60 auth_method 0 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 10 monclient(hunting): handle_auth_done global_id 27933 payload 824 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93e408e7a0 con 0x7f93e406fd70 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e8f86700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 0x7f93e402b6a0 secure :-1 s=READY pgs=6586 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f60 tx=0x7f93e0001e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 client.22151.objecter ms_handle_connect 0x7f93e406fd70 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 client.22151.objecter resend_mon_ops 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0005b70 con 0x7f93e406fd70 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 monclient: dump: 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.194 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 client.22151.objecter ms_handle_connect 0x7f93c8036f60 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93e9f88700 1 --2- 172.21.15.5:0/1065114412 >> v2:172.21.15.143:6812/1098205034 conn(0x7f93c8036f60 0x7f93c8039420 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009d90 tx=0x7f93dc011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f93e0005cd0 con 0x7f93e406fd70 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:59.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 msgr2=0x7f93e402b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 0x7f93e402b6a0 secure :-1 s=READY pgs=6586 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f60 tx=0x7f93e0001e60 comp rx=0 tx=0).stop 2024-09-16T10:51:59.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 msgr2=0x7f93e402b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.172+0000 7f93d9ffb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e406fd70 0x7f93e402b6a0 secure :-1 s=CLOSED pgs=6586 cs=0 l=1 rev1=1 crypto rx=0x7f93e0000f60 tx=0x7f93e0001e60 comp rx=0 tx=0).stop 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93eb9eb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 0x7f93e402bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93eb9eb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e404efc0 con 0x7f93e4070150 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 0x7f93e402bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient: get_auth_request con 0x7f93e4070150 auth_method 0 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 10 monclient(hunting): handle_auth_done global_id 27936 payload 824 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 1 -- 172.21.15.5:0/1065114412 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f93e40a6df0 con 0x7f93e4070150 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 client.22151.objecter ms_handle_connect 0x7f93e4070150 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 client.22151.objecter resend_mon_ops 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93e8f86700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 0x7f93e402bbe0 secure :-1 s=READY pgs=6587 cs=0 l=1 rev1=1 crypto rx=0x7f93e0017000 tx=0x7f93e0001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0023070 con 0x7f93e4070150 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 monclient: dump: 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.199 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f93e000dd00 con 0x7f93e4070150 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 msgr2=0x7f93e402bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 0x7f93e402bbe0 secure :-1 s=READY pgs=6587 cs=0 l=1 rev1=1 crypto rx=0x7f93e0017000 tx=0x7f93e0001150 comp rx=0 tx=0).stop 2024-09-16T10:51:59.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 msgr2=0x7f93e402bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.184+0000 7f93d9ffb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93e4070150 0x7f93e402bbe0 secure :-1 s=CLOSED pgs=6587 cs=0 l=1 rev1=1 crypto rx=0x7f93e0017000 tx=0x7f93e0001150 comp rx=0 tx=0).stop 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:51:59.201 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.202 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.203 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:51:59.204 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:51:59.205 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.206 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:51:59.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:51:59.209 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.211 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.212 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.213 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.214 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.215 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.215 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:51:59.215 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:51:59.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 10 librados: watch_flush enter 2024-09-16T10:51:59.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 10 librados: watch_flush exit 2024-09-16T10:51:59.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 20 client.22151.objecter shutdown clearing up homeless session... 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 10 client.22151.objecter successfully canceled tick 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 >> v2:172.21.15.143:6812/1098205034 conn(0x7f93c8036f60 msgr2=0x7f93c8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1065114412 >> v2:172.21.15.143:6812/1098205034 conn(0x7f93c8036f60 0x7f93c8039420 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009d90 tx=0x7f93dc011000 comp rx=0 tx=0).stop 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 10 monclient: shutdown 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 msgr2=0x7f93e41ceac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 secure :-1 s=READY pgs=4490 cs=0 l=1 rev1=1 crypto rx=0x7f93d400e7e0 tx=0x7f93d400eaf0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93e9f88700 1 -- 172.21.15.5:0/1065114412 reap_dead start 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 shutdown_connections 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 --2- 172.21.15.5:0/1065114412 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f93e40a2eb0 0x7f93e41ceac0 unknown :-1 s=CLOSED pgs=4490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 >> 172.21.15.5:0/1065114412 conn(0x7f93e40a4fb0 msgr2=0x7f93e413bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 shutdown_connections 2024-09-16T10:51:59.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 -- 172.21.15.5:0/1065114412 wait complete. 2024-09-16T10:51:59.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.188+0000 7f93eb9eb700 1 librados: shutdown 2024-09-16T10:51:59.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:51:59.218 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:51:59.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: get_monmap_and_config 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: monmap: 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:59.342910+0000 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:59.342910+0000 2024-09-16T10:51:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:59.344 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 Processor -- start 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 -- start start 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: init 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f516c09b3e0 0x7f516c09b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f516c09b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f516c09bd00 0x7f516c09c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f516c09bd00 addr v2:172.21.15.143:3300/0 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f516c1326b0 addr v1:172.21.15.143:6789/0 2024-09-16T10:51:59.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f516c09fd30 con 0x7f516c1326b0 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c09e350 con 0x7f516c09b3e0 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c09ca20 con 0x7f516c09bd00 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516b59e700 10 monclient(hunting): authenticate will time out at 977.919660s 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f516c09b3e0 0x7f516c09b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f516c09b3e0 0x7f516c09b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38314/0 (socket says 172.21.15.5:38314) 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 1 -- 172.21.15.5:0/1789143371 learned_addr learned my addr 172.21.15.5:0/1789143371 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 10 monclient(hunting): get_auth_request con 0x7f516c09b3e0 auth_method 0 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.340+0000 7f516a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f5169d9b700 1 --2- 172.21.15.5:0/1789143371 >> v2:172.21.15.143:3300/0 conn(0x7f516c09bd00 0x7f516c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f5169d9b700 10 monclient(hunting): get_auth_request con 0x7f516c09bd00 auth_method 0 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f5169d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f5169d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f5169d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient(hunting): handle_auth_done global_id 27939 payload 274 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 -- 172.21.15.5:0/1789143371 >> v2:172.21.15.143:3300/0 conn(0x7f516c09bd00 msgr2=0x7f516c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 --2- 172.21.15.5:0/1789143371 >> v2:172.21.15.143:3300/0 conn(0x7f516c09bd00 0x7f516c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 -- 172.21.15.5:0/1789143371 >> v1:172.21.15.143:6789/0 conn(0x7f516c1326b0 legacy=0x7f516c132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 monclient: found mon.noname-a 2024-09-16T10:51:59.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:51:59.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 -- 172.21.15.5:0/1789143371 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c09cd00 con 0x7f516c09b3e0 2024-09-16T10:51:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516a59c700 1 --2- 172.21.15.5:0/1789143371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c09b7c0 secure :-1 s=READY pgs=6588 cs=0 l=1 rev1=1 crypto rx=0x7f515c009e50 tx=0x7f515c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16188a3071cf1bd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 1 -- 172.21.15.5:0/1789143371 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00e070 con 0x7f516c09b3e0 2024-09-16T10:51:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:51:59.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: dump: 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 5 monclient: authenticate success, global_id 27939 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 1 -- 172.21.15.5:0/1789143371 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00b980 con 0x7f516c09b3e0 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 4 set_mon_vals no callback set 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 1 -- 172.21.15.5:0/1789143371 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00bd30 con 0x7f516c09b3e0 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:51:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516959a700 10 monclient: dump: 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 10 monclient: shutdown 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 -- 172.21.15.5:0/1789143371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 msgr2=0x7f516c09b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 --2- 172.21.15.5:0/1789143371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c09b7c0 secure :-1 s=READY pgs=6588 cs=0 l=1 rev1=1 crypto rx=0x7f515c009e50 tx=0x7f515c00b330 comp rx=0 tx=0).stop 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 -- 172.21.15.5:0/1789143371 shutdown_connections 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 --2- 172.21.15.5:0/1789143371 >> v2:172.21.15.143:3300/0 conn(0x7f516c09bd00 0x7f516c09c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 --2- 172.21.15.5:0/1789143371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c09b7c0 unknown :-1 s=CLOSED pgs=6588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 -- 172.21.15.5:0/1789143371 >> 172.21.15.5:0/1789143371 conn(0x7f516c0a45d0 msgr2=0x7f516c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 -- 172.21.15.5:0/1789143371 shutdown_connections 2024-09-16T10:51:59.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 -- 172.21.15.5:0/1789143371 wait complete. 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 10 monclient: monmap: 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 librados: starting msgr at 2024-09-16T10:51:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 librados: starting objecter 2024-09-16T10:51:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.344+0000 7f516b59e700 1 Processor -- start 2024-09-16T10:51:59.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- start start 2024-09-16T10:51:59.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 librados: setting wanted keys 2024-09-16T10:51:59.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 librados: calling monclient init 2024-09-16T10:51:59.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: init 2024-09-16T10:51:59.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c1ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f516c09b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c09bd00 0x7f516c131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f516c09bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c1326b0 0x7f516c1c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f516c1326b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c1260d0 con 0x7f516c09b3e0 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c123c20 con 0x7f516c1326b0 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c09b0d0 con 0x7f516c09bd00 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient(hunting): authenticate will time out at 977.925172s 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c1ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c1ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38326/0 (socket says 172.21.15.5:38326) 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 -- 172.21.15.5:0/3739863488 learned_addr learned my addr 172.21.15.5:0/3739863488 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): get_auth_request con 0x7f516c09b3e0 auth_method 0 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c09bd00 0x7f516c131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): get_auth_request con 0x7f516c09bd00 auth_method 0 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516ad9d700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c1326b0 0x7f516c1c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516ad9d700 10 monclient(hunting): get_auth_request con 0x7f516c1326b0 auth_method 0 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient(hunting): handle_auth_done global_id 27942 payload 824 2024-09-16T10:51:59.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c09bd00 msgr2=0x7f516c131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c09bd00 0x7f516c131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c1326b0 msgr2=0x7f516c1c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c1326b0 0x7f516c1c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 monclient: found mon.noname-a 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:51:59.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c009b30 con 0x7f516c09b3e0 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516a59c700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c1ca820 secure :-1 s=READY pgs=6589 cs=0 l=1 rev1=1 crypto rx=0x7f515c00b370 tx=0x7f515c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 client.?.objecter ms_handle_connect 0x7f516c09b3e0 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c004930 con 0x7f516c09b3e0 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.375 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c0092e0 con 0x7f516c09b3e0 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c01c680 con 0x7f516c09b3e0 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 5 monclient: authenticate success, global_id 27942 2024-09-16T10:51:59.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _renew_subs 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c070c00 con 0x7f516c09b3e0 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5163fff700 4 set_mon_vals no callback set 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.378 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 client.27942.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _renew_subs 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c125df0 con 0x7f516c09b3e0 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.348+0000 7f516b59e700 1 librados: init done 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516b59e700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 0x7f516c1c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c067380 con 0x7f516c1321a0 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 0x7f516c1c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient: get_auth_request con 0x7f516c1321a0 auth_method 0 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 10 monclient(hunting): handle_auth_done global_id 22157 payload 824 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f515c01c7e0 con 0x7f516c09b3e0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f516c070dd0 con 0x7f516c1321a0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5169d9b700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 0x7f516c1c7860 secure :-1 s=READY pgs=4491 cs=0 l=1 rev1=1 crypto rx=0x7f5154011800 tx=0x7f5154011bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 --2- 172.21.15.5:0/3739863488 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5150036f60 0x7f5150039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 client.27942.objecter ms_handle_connect 0x7f516c1321a0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 client.27942.objecter resend_mon_ops 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7334+0+0 (secure 0 0 0) 0x7f515c049e80 con 0x7f516c09b3e0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 client.27942.objecter ms_dispatch 0x7f516c000fd0 osd_map(68..68 src has 1..68) v4 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 3 client.27942.objecter handle_osd_map got epochs [68,68] > 0 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 3 client.27942.objecter handle_osd_map decoding full epoch 68 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 20 client.27942.objecter dump_active .. 0 homeless 2024-09-16T10:51:59.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51540045a0 con 0x7f516c1321a0 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.382 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516ad9d700 1 --2- 172.21.15.5:0/3739863488 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5150036f60 0x7f5150039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516ad9d700 10 monclient: get_auth_request con 0x7f5150036f60 auth_method 0 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f51540036c0 con 0x7f516c1321a0 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 msgr2=0x7f516c1c7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 0x7f516c1c7860 secure :-1 s=READY pgs=4491 cs=0 l=1 rev1=1 crypto rx=0x7f5154011800 tx=0x7f5154011bc0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 msgr2=0x7f516c1c7860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1321a0 0x7f516c1c7860 secure :-1 s=CLOSED pgs=4491 cs=0 l=1 rev1=1 crypto rx=0x7f5154011800 tx=0x7f5154011bc0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f516ad9d700 1 --2- 172.21.15.5:0/3739863488 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5150036f60 0x7f5150039420 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f5164001640 tx=0x7f5164009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.352+0000 7f5162ffd700 10 client.27942.objecter ms_handle_connect 0x7f5150036f60 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f516b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:51:59.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f516b59e700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 0x7f516c11a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f516b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c09a2e0 con 0x7f516c1c7da0 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 0x7f516c11a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient: get_auth_request con 0x7f516c1c7da0 auth_method 0 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 10 monclient(hunting): handle_auth_done global_id 22160 payload 824 2024-09-16T10:51:59.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.360+0000 7f5169d9b700 1 -- 172.21.15.5:0/3739863488 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f516c09ff00 con 0x7f516c1c7da0 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 client.27942.objecter ms_handle_connect 0x7f516c1c7da0 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 client.27942.objecter resend_mon_ops 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5169d9b700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 0x7f516c11a530 secure :-1 s=READY pgs=4492 cs=0 l=1 rev1=1 crypto rx=0x7f5154011800 tx=0x7f5154004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5154025070 con 0x7f516c1c7da0 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.385 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f5154015460 con 0x7f516c1c7da0 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 msgr2=0x7f516c11a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 0x7f516c11a530 secure :-1 s=READY pgs=4492 cs=0 l=1 rev1=1 crypto rx=0x7f5154011800 tx=0x7f5154004770 comp rx=0 tx=0).stop 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 msgr2=0x7f516c11a530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f5162ffd700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f516c1c7da0 0x7f516c11a530 unknown :-1 s=CLOSED pgs=4492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:51:59.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.388 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.389 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:51:59.390 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.391 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.392 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.393 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:51:59.394 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.395 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.397 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.398 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.399 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.400 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:51:59.401 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 10 librados: watch_flush enter 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 10 librados: watch_flush exit 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 20 client.27942.objecter shutdown clearing up homeless session... 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 10 client.27942.objecter successfully canceled tick 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5150036f60 msgr2=0x7f5150039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 --2- 172.21.15.5:0/3739863488 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5150036f60 0x7f5150039420 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f5164001640 tx=0x7f5164009040 comp rx=0 tx=0).stop 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 10 monclient: shutdown 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 msgr2=0x7f516c1ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 --2- 172.21.15.5:0/3739863488 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f516c09b3e0 0x7f516c1ca820 secure :-1 s=READY pgs=6589 cs=0 l=1 rev1=1 crypto rx=0x7f515c00b370 tx=0x7f515c004af0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516ad9d700 1 -- 172.21.15.5:0/3739863488 reap_dead start 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516ad9d700 1 -- 172.21.15.5:0/3739863488 reap_dead start 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 shutdown_connections 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 >> 172.21.15.5:0/3739863488 conn(0x7f516c0a45d0 msgr2=0x7f516c0a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 shutdown_connections 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 -- 172.21.15.5:0/3739863488 wait complete. 2024-09-16T10:51:59.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.364+0000 7f516b59e700 1 librados: shutdown 2024-09-16T10:51:59.405 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:51:59.405 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:51:59.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.528+0000 7f7ad5aec700 10 monclient: get_monmap_and_config 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.528+0000 7f7ad5aec700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.528+0000 7f7ad5aec700 10 monclient: monmap: 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:59.532734+0000 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:59.532734+0000 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.533 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:59.534 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:59.534 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 Processor -- start 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- start start 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient: init 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-b con 0x7f7ad00a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ad009f780 addr v1:172.21.15.143:6789/0 2024-09-16T10:51:59.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ad00a3510 0x7f7ad01360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-c con 0x7f7ad00a3510 addr v2:172.21.15.143:3300/0 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad00a3c40 con 0x7f7ad009f780 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad00a4300 con 0x7f7ad00a3510 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad009fda0 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient(hunting): authenticate will time out at 978.109226s 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46620/0 (socket says 172.21.15.5:46620) 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 -- 172.21.15.5:0/271526296 learned_addr learned my addr 172.21.15.5:0/271526296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request con 0x7f7ad00a2bf0 auth_method 0 2024-09-16T10:51:59.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 1 --2- 172.21.15.5:0/271526296 >> v2:172.21.15.143:3300/0 conn(0x7f7ad00a3510 0x7f7ad01360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): get_auth_request con 0x7f7ad00a3510 auth_method 0 2024-09-16T10:51:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_done global_id 22163 payload 274 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 -- 172.21.15.5:0/271526296 >> v2:172.21.15.143:3300/0 conn(0x7f7ad00a3510 msgr2=0x7f7ad01360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/271526296 >> v2:172.21.15.143:3300/0 conn(0x7f7ad00a3510 0x7f7ad01360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 -- 172.21.15.5:0/271526296 >> v1:172.21.15.143:6789/0 conn(0x7f7ad009f780 legacy=0x7f7ad01359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 monclient: found mon.noname-b 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 -- 172.21.15.5:0/271526296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad00a0080 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/271526296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 secure :-1 s=READY pgs=4493 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009910 tx=0x7f7ac4009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af04face23d3e5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 1 -- 172.21.15.5:0/271526296 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4011830 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: dump: 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 5 monclient: authenticate success, global_id 22163 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 1 -- 172.21.15.5:0/271526296 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4011990 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 4 set_mon_vals no callback set 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 1 -- 172.21.15.5:0/271526296 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4011d40 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7acf7fe700 10 monclient: dump: 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.544 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 10 monclient: shutdown 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- 172.21.15.5:0/271526296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 msgr2=0x7f7ad00a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/271526296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 secure :-1 s=READY pgs=4493 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009910 tx=0x7f7ac4009c20 comp rx=0 tx=0).stop 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- 172.21.15.5:0/271526296 shutdown_connections 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/271526296 >> v2:172.21.15.143:3300/0 conn(0x7f7ad00a3510 0x7f7ad01360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/271526296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad00a2fd0 unknown :-1 s=CLOSED pgs=4493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- 172.21.15.5:0/271526296 >> 172.21.15.5:0/271526296 conn(0x7f7ad00a53e0 msgr2=0x7f7ad0073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- 172.21.15.5:0/271526296 shutdown_connections 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.532+0000 7f7ad5aec700 1 -- 172.21.15.5:0/271526296 wait complete. 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: monmap: 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 librados: starting msgr at 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 librados: starting objecter 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 Processor -- start 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- start start 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 librados: setting wanted keys 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 librados: calling monclient init 2024-09-16T10:51:59.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: init 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad009f780 0x7f7ad0135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-b con 0x7f7ad009f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad01362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-c con 0x7f7ad00a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ad00a3510 0x7f7ad011f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): picked mon.noname-a con 0x7f7ad00a3510 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01cb590 con 0x7f7ad00a3510 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0122e80 con 0x7f7ad009f780 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad01367e0 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient(hunting): authenticate will time out at 978.114192s 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad009f780 0x7f7ad0135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad01362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ad00a3510 0x7f7ad011f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad009f780 0x7f7ad0135d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:33198/0 (socket says 172.21.15.5:33198) 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): get_auth_request con 0x7f7ad00a2bf0 auth_method 0 2024-09-16T10:51:59.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 learned_addr learned my addr 172.21.15.5:0/3854796396 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request con 0x7f7ad009f780 auth_method 0 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): get_auth_request con 0x7f7ad00a3510 auth_method 0 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad52eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient(hunting): handle_auth_done global_id 22166 payload 824 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad009f780 msgr2=0x7f7ad0135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad009f780 0x7f7ad0135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ad00a3510 msgr2=0x7f7ad011f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ad00a3510 0x7f7ad011f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 monclient: found mon.noname-c 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac4009570 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acffff700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad01362a0 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000cbb0 tx=0x7f7ac000cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 client.?.objecter ms_handle_connect 0x7f7ad00a2bf0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0003c20 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: dump: 2024-09-16T10:51:59.559 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 5 monclient: authenticate success, global_id 22166 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac0003d80 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000aaa0 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7acd7fa700 10 monclient: dump: 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _renew_subs 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7aceffd700 4 set_mon_vals no callback set 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad0123720 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 client.22166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _renew_subs 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad0123440 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.536+0000 7f7ad5aec700 1 librados: init done 2024-09-16T10:51:59.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad5aec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0074d50 con 0x7f7ad01ca830 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7ac000ac00 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient: get_auth_request con 0x7f7ad01ca830 auth_method 0 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 0x7f7abc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_done global_id 34138 payload 824 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ad00a3940 con 0x7f7ad01ca830 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7334+0+0 (secure 0 0 0) 0x7f7ac00485b0 con 0x7f7ad00a2bf0 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter ms_dispatch 0x7f7ad0000fd0 osd_map(68..68 src has 1..68) v4 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 3 client.22166.objecter handle_osd_map got epochs [68,68] > 0 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 3 client.22166.objecter handle_osd_map decoding full epoch 68 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 20 client.22166.objecter dump_active .. 0 homeless 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad52eb700 1 --2- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 0x7f7abc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter ms_handle_connect 0x7f7ad01ca830 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter resend_mon_ops 2024-09-16T10:51:59.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad52eb700 10 monclient: get_auth_request con 0x7f7abc036f60 auth_method 0 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 msgr2=0x7f7ad011fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 msgr2=0x7f7ad011fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac4004770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4000c00 tx=0x7f7ac4004770 comp rx=0 tx=0).stop 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 msgr2=0x7f7ad011fcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 0x7f7abc0454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 msgr2=0x7f7ad011fcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01ca830 0x7f7ad011fcb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc045b00 con 0x7f7abc043060 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 0x7f7abc0454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient: get_auth_request con 0x7f7abc043060 auth_method 0 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad52eb700 1 --2- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 0x7f7abc039420 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009bf0 tx=0x7f7ac8009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter ms_handle_connect 0x7f7abc036f60 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_done global_id 34141 payload 824 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7abc045c70 con 0x7f7abc043060 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter ms_handle_connect 0x7f7abc043060 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 0x7f7abc0454a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009c60 tx=0x7f7ac4011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 client.22166.objecter resend_mon_ops 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4020070 con 0x7f7abc043060 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.540+0000 7f7acd7fa700 10 monclient: dump: 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.567 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f7ac4004620 con 0x7f7abc043060 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 msgr2=0x7f7abc0454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 0x7f7abc0454a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009c60 tx=0x7f7ac4011000 comp rx=0 tx=0).stop 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 msgr2=0x7f7abc0454a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.544+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7abc043060 0x7f7abc0454a0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009c60 tx=0x7f7ac4011000 comp rx=0 tx=0).stop 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad5aec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 0x7f7ad006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad5aec700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad0136920 con 0x7f7ad01201f0 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 0x7f7ad006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient: get_auth_request con 0x7f7ad01201f0 auth_method 0 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 10 monclient(hunting): handle_auth_done global_id 34144 payload 824 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 1 -- 172.21.15.5:0/3854796396 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7ad0120600 con 0x7f7ad01201f0 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 10 client.22166.objecter ms_handle_connect 0x7f7ad01201f0 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7ad4aea700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 0x7f7ad006cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7abc045c40 tx=0x7f7ac4028000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 10 client.22166.objecter resend_mon_ops 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4047070 con 0x7f7ad01201f0 2024-09-16T10:51:59.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.556+0000 7f7acd7fa700 10 monclient: dump: 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f7ac403f040 con 0x7f7ad01201f0 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 msgr2=0x7f7ad006cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 0x7f7ad006cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7abc045c40 tx=0x7f7ac4028000 comp rx=0 tx=0).stop 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 msgr2=0x7f7ad006cd60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad52eb700 1 -- 172.21.15.5:0/3854796396 reap_dead start 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7acd7fa700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ad01201f0 0x7f7ad006cd60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:51:59.573 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:51:59.574 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:51:59.575 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:51:59.576 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:51:59.577 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.578 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.579 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.580 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:51:59.581 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:51:59.582 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.584 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:51:59.585 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 10 librados: watch_flush enter 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 10 librados: watch_flush exit 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 20 client.22166.objecter shutdown clearing up homeless session... 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 10 client.22166.objecter successfully canceled tick 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 msgr2=0x7f7abc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 0x7f7abc039420 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009bf0 tx=0x7f7ac8009240 comp rx=0 tx=0).stop 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 10 monclient: shutdown 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 msgr2=0x7f7ad01362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad01362a0 secure :-1 s=READY pgs=4494 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000cbb0 tx=0x7f7ac000cec0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 shutdown_connections 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7abc036f60 0x7f7abc039420 unknown :-1 s=CLOSED pgs=3660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 --2- 172.21.15.5:0/3854796396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7ad00a2bf0 0x7f7ad01362a0 unknown :-1 s=CLOSED pgs=4494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 >> 172.21.15.5:0/3854796396 conn(0x7f7ad00a53e0 msgr2=0x7f7ad00a60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 shutdown_connections 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 -- 172.21.15.5:0/3854796396 wait complete. 2024-09-16T10:51:59.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.560+0000 7f7ad5aec700 1 librados: shutdown 2024-09-16T10:51:59.589 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:51:59.589 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:51:59.590 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:51:59.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.720+0000 7fca8e528700 10 monclient: get_monmap_and_config 2024-09-16T10:51:59.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.720+0000 7fca8e528700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.720+0000 7fca8e528700 10 monclient: monmap: 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:51:59.724397+0000 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:51:59.724397+0000 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:51:59.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:51:59.726 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:51:59.726 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:51:59.726 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 Processor -- start 2024-09-16T10:51:59.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- start start 2024-09-16T10:51:59.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: init 2024-09-16T10:51:59.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca880a2bf0 0x7fca880a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-c con 0x7fca880a2bf0 addr v2:172.21.15.143:3300/0 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca8809f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 0x7fca881360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-b con 0x7fca880a3510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880a3c40 con 0x7fca8809f780 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca880a4300 con 0x7fca880a2bf0 2024-09-16T10:51:59.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8809fda0 con 0x7fca880a3510 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient(hunting): authenticate will time out at 978.300898s 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8cd25700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fca8809f780 0x7fca881359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51414/0 (socket says 172.21.15.5:51414) 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8cd25700 1 -- 172.21.15.5:0/2767732049 learned_addr learned my addr 172.21.15.5:0/2767732049 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 --2- 172.21.15.5:0/2767732049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 0x7fca881360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): get_auth_request con 0x7fca880a3510 auth_method 0 2024-09-16T10:51:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 1 --2- 172.21.15.5:0/2767732049 >> v2:172.21.15.143:3300/0 conn(0x7fca880a2bf0 0x7fca880a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): get_auth_request con 0x7fca880a2bf0 auth_method 0 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 1 -- 172.21.15.5:0/2767732049 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212050810 0 0) 0x7fca880a3c40 con 0x7fca8809f780 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient(hunting): _negotiate 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient(hunting): my global_id is 27948 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 1 -- 172.21.15.5:0/2767732049 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca740035d0 con 0x7fca8809f780 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient(hunting): handle_auth_done global_id 22169 payload 274 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 -- 172.21.15.5:0/2767732049 >> v2:172.21.15.143:3300/0 conn(0x7fca880a2bf0 msgr2=0x7fca880a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 --2- 172.21.15.5:0/2767732049 >> v2:172.21.15.143:3300/0 conn(0x7fca880a2bf0 0x7fca880a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 -- 172.21.15.5:0/2767732049 >> v1:172.21.15.5:6789/0 conn(0x7fca8809f780 legacy=0x7fca881359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 monclient: found mon.noname-b 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 -- 172.21.15.5:0/2767732049 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880a0080 con 0x7fca880a3510 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8dd27700 1 --2- 172.21.15.5:0/2767732049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 0x7fca881360c0 secure :-1 s=READY pgs=4495 cs=0 l=1 rev1=1 crypto rx=0x7fca80009d90 tx=0x7fca8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4accd7d9f1ac4657 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 1 -- 172.21.15.5:0/2767732049 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8000e070 con 0x7fca880a3510 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: dump: 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.743 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 5 monclient: authenticate success, global_id 22169 2024-09-16T10:51:59.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 1 -- 172.21.15.5:0/2767732049 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8000bb20 con 0x7fca880a3510 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: get_monmap_and_config success 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 4 set_mon_vals no callback set 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 1 -- 172.21.15.5:0/2767732049 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80012430 con 0x7fca880a3510 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca7ffff700 10 monclient: dump: 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: shutdown 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- 172.21.15.5:0/2767732049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 msgr2=0x7fca881360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 --2- 172.21.15.5:0/2767732049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 0x7fca881360c0 secure :-1 s=READY pgs=4495 cs=0 l=1 rev1=1 crypto rx=0x7fca80009d90 tx=0x7fca8000b330 comp rx=0 tx=0).stop 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- 172.21.15.5:0/2767732049 shutdown_connections 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 --2- 172.21.15.5:0/2767732049 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a3510 0x7fca881360c0 unknown :-1 s=CLOSED pgs=4495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 --2- 172.21.15.5:0/2767732049 >> v2:172.21.15.143:3300/0 conn(0x7fca880a2bf0 0x7fca880a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- 172.21.15.5:0/2767732049 >> 172.21.15.5:0/2767732049 conn(0x7fca880a53e0 msgr2=0x7fca88073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- 172.21.15.5:0/2767732049 shutdown_connections 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- 172.21.15.5:0/2767732049 wait complete. 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: build_initial_monmap 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: monmap: 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:51:59.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 librados: starting msgr at 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 librados: starting objecter 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 Processor -- start 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 -- start start 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 librados: setting wanted keys 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 1 librados: calling monclient init 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.724+0000 7fca8e528700 10 monclient: init 2024-09-16T10:51:59.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _reopen_session rank -1 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-a con 0x7fca8809f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-c con 0x7fca880a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca880a3510 0x7fca881364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): picked mon.noname-b con 0x7fca880a3510 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca881ca8d0 con 0x7fca8809f780 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8809f4e0 con 0x7fca880a3510 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): start opening mon connection 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca88070460 con 0x7fca880a2bf0 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): _renew_subs 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient(hunting): authenticate will time out at 978.304272s 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38354/0 (socket says 172.21.15.5:38354) 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46644/0 (socket says 172.21.15.5:46644) 2024-09-16T10:51:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 -- 172.21.15.5:0/3316744125 learned_addr learned my addr 172.21.15.5:0/3316744125 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): get_auth_request con 0x7fca8809f780 auth_method 0 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): get_auth_request con 0x7fca880a2bf0 auth_method 0 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca880a3510 0x7fca881364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): get_auth_request con 0x7fca880a3510 auth_method 0 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:51:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8dd27700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient(hunting): handle_auth_done global_id 22172 payload 824 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 -- 172.21.15.5:0/3316744125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca880a3510 msgr2=0x7fca881364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca880a3510 0x7fca881364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 -- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 msgr2=0x7fca881ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient: _finish_hunting 0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 monclient: found mon.noname-c 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 -- 172.21.15.5:0/3316744125 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80009a70 con 0x7fca880a2bf0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8cd25700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 secure :-1 s=READY pgs=4496 cs=0 l=1 rev1=1 crypto rx=0x7fca7800c500 tx=0x7fca7800c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 client.?.objecter ms_handle_connect 0x7fca880a2bf0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78005d90 con 0x7fca880a2bf0 2024-09-16T10:51:59.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: dump: 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: _finish_auth 0 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca780070d0 con 0x7fca880a2bf0 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 5 monclient: authenticate success, global_id 22172 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _renew_subs 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca880a78e0 con 0x7fca880a2bf0 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78007440 con 0x7fca880a2bf0 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:51:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 monclient: dump: 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7f7fe700 4 set_mon_vals no callback set 2024-09-16T10:51:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 client.22172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _renew_subs 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca881236f0 con 0x7fca880a2bf0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 librados: init done 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fca780075a0 con 0x7fca880a2bf0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca880a2fd0 con 0x7fca880a2bf0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 --2- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 0x7fca6c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7723+0+0 (secure 0 0 0) 0x7fca78047a50 con 0x7fca880a2bf0 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 client.22172.objecter ms_dispatch 0x7fca88000fd0 osd_map(69..69 src has 1..69) v4 2024-09-16T10:51:59.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 3 client.22172.objecter handle_osd_map got epochs [69,69] > 0 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 3 client.22172.objecter handle_osd_map decoding full epoch 69 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 20 client.22172.objecter dump_active .. 0 homeless 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 --2- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 0x7fca6c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 10 monclient: get_auth_request con 0x7fca6c036f60 auth_method 0 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca7dffb700 10 client.22172.objecter ms_handle_connect 0x7fca6c036f60 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.728+0000 7fca8d526700 1 --2- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 0x7fca6c039420 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7fca840113d0 tx=0x7fca84009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.732+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fca78019b40 con 0x7fca880a2bf0 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.732+0000 7fca7dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:51:59.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.732+0000 7fca7dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:51:59.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.972+0000 7fca8e528700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:51:59.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.972+0000 7fca8e528700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:51:59.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.972+0000 7fca8e528700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:51:59.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.972+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fca88121f10 con 0x7fca880a2bf0 2024-09-16T10:51:59.977 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.978 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=1249465958,config=692619927,health=3914089016,kv=564032378,logm=966266958}) 2024-09-16T10:51:59.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca7dffb700 1 -- 172.21.15.5:0/3316744125 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fca78017020 con 0x7fca880a2bf0 2024-09-16T10:51:59.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca7dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:51:59.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca7dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:51:59.979 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=342460059,config=692619927,health=3547182302,kv=2393611736,logm=809149927}) 2024-09-16T10:51:59.979 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.979 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=1249465958,config=692619927,health=3914089016,kv=564032378,logm=966266958}) 2024-09-16T10:51:59.979 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=10,kv=10,logm=70} crc {auth=3930604977,config=692619927,health=2253386788,kv=564032378,logm=2731657830}) 2024-09-16T10:51:59.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 10 librados: watch_flush enter 2024-09-16T10:51:59.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 10 librados: watch_flush exit 2024-09-16T10:51:59.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 20 client.22172.objecter shutdown clearing up homeless session... 2024-09-16T10:51:59.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 10 client.22172.objecter successfully canceled tick 2024-09-16T10:51:59.980 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.981 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892968505}) 2024-09-16T10:51:59.981 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065900956}) 2024-09-16T10:51:59.981 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 msgr2=0x7fca6c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 0x7fca6c039420 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7fca840113d0 tx=0x7fca84009240 comp rx=0 tx=0).stop 2024-09-16T10:51:59.981 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892968505}) 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 10 monclient: shutdown 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 msgr2=0x7fca88135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 secure :-1 s=READY pgs=4496 cs=0 l=1 rev1=1 crypto rx=0x7fca7800c500 tx=0x7fca7800c810 comp rx=0 tx=0).stop 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 shutdown_connections 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca880a3510 0x7fca881364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fca880a2bf0 0x7fca88135f60 unknown :-1 s=CLOSED pgs=4496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> v2:172.21.15.143:6812/1098205034 conn(0x7fca6c036f60 0x7fca6c039420 unknown :-1 s=CLOSED pgs=3661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.983 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.976+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481870547}) 2024-09-16T10:51:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 --2- 172.21.15.5:0/3316744125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fca8809f780 0x7fca881ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:51:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 >> 172.21.15.5:0/3316744125 conn(0x7fca880a53e0 msgr2=0x7fca880a60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:51:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 shutdown_connections 2024-09-16T10:51:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 -- 172.21.15.5:0/3316744125 wait complete. 2024-09-16T10:51:59.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:51:59.976+0000 7fca8e528700 1 librados: shutdown 2024-09-16T10:51:59.984 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.984 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291400586}) 2024-09-16T10:51:59.984 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1263949135}) 2024-09-16T10:51:59.984 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.984 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291400586}) 2024-09-16T10:51:59.985 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.980+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513337293}) 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338287100}) 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1941313864}) 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338287100}) 2024-09-16T10:51:59.986 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133587602}) 2024-09-16T10:51:59.987 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.987 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501747025}) 2024-09-16T10:51:59.987 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513208775}) 2024-09-16T10:51:59.987 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.988 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501747025}) 2024-09-16T10:51:59.988 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.984+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=924813853}) 2024-09-16T10:51:59.989 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.989 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077207106}) 2024-09-16T10:51:59.989 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3250011733}) 2024-09-16T10:51:59.989 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.990 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077207106}) 2024-09-16T10:51:59.990 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914108192}) 2024-09-16T10:51:59.990 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.991 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191171261}) 2024-09-16T10:51:59.991 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159424694}) 2024-09-16T10:51:59.991 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.991 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191171261}) 2024-09-16T10:51:59.991 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090269961}) 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087007874}) 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111064658}) 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087007874}) 2024-09-16T10:51:59.992 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.988+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705435533}) 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100824397}) 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602597284}) 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100824397}) 2024-09-16T10:51:59.994 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2398863431}) 2024-09-16T10:51:59.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=973130748}) 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087843046}) 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=973130748}) 2024-09-16T10:51:59.996 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.992+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525079942}) 2024-09-16T10:51:59.997 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.997 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=1} crc {logm=3446517368,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=555153161,mon_config_key=4120093134,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=502782655,osd_snap=1820152508,osdmap=824841097}) 2024-09-16T10:51:59.997 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=3} crc {logm=2291997015,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=867380045,mon_config_key=70326691,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=502782655,osd_snap=1820152508,osdmap=2534363998}) 2024-09-16T10:51:59.997 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:51:59.998 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=1} crc {logm=3446517368,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=555153161,mon_config_key=4120093134,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=502782655,osd_snap=1820152508,osdmap=824841097}) 2024-09-16T10:51:59.998 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18,osdmap=3} crc {logm=2888452794,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=555153161,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=502782655,osd_snap=3448247511,osdmap=842491992}) 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2033460634}) 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3282347948}) 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2033460634}) 2024-09-16T10:52:00.000 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:51:59.996+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1109936730}) 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=420284380}) 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=13} crc {osdmap=3607296179}) 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=420284380}) 2024-09-16T10:52:00.004 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:52:00.000+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=2584223162}) 2024-09-16T10:52:00.997 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:52:00.997 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:52:01.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 10 monclient: get_monmap_and_config 2024-09-16T10:52:01.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 10 monclient: monmap: 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:01.115631+0000 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:01.115631+0000 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:01.116 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 1 Processor -- start 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 1 -- start start 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.112+0000 7f5b8aaea700 10 monclient: init 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b84070380 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5b84070760 0x7f5b8413a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-c con 0x7f5b84070760 addr v2:172.21.15.143:3300/0 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5b84070bf0 0x7f5b8413ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-a con 0x7f5b84070bf0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b84072940 con 0x7f5b84070380 2024-09-16T10:52:01.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b8406cb50 con 0x7f5b84070bf0 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b84135de0 con 0x7f5b84070760 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient(hunting): authenticate will time out at 979.692494s 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5b84070bf0 0x7f5b8413ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b89ae8700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5b84070380 0x7f5b841356d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:37414/0 (socket says 172.21.15.5:37414) 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5b84070bf0 0x7f5b8413ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38366/0 (socket says 172.21.15.5:38366) 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b89ae8700 1 -- 172.21.15.5:0/3976242540 learned_addr learned my addr 172.21.15.5:0/3976242540 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5b84070760 0x7f5b8413a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): get_auth_request con 0x7f5b84070bf0 auth_method 0 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): get_auth_request con 0x7f5b84070760 auth_method 0 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b892e7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient(hunting): handle_auth_done global_id 27960 payload 274 2024-09-16T10:52:01.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 1 -- 172.21.15.5:0/3976242540 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601422503 0 0) 0x7f5b84072940 con 0x7f5b84070380 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 -- 172.21.15.5:0/3976242540 >> v2:172.21.15.143:3300/0 conn(0x7f5b84070760 msgr2=0x7f5b8413a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 --2- 172.21.15.5:0/3976242540 >> v2:172.21.15.143:3300/0 conn(0x7f5b84070760 0x7f5b8413a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 -- 172.21.15.5:0/3976242540 >> v1:172.21.15.143:6789/0 conn(0x7f5b84070380 legacy=0x7f5b841356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 monclient: found mon.noname-a 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 -- 172.21.15.5:0/3976242540 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8409eed0 con 0x7f5b84070bf0 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:52:01.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8a2e9700 1 --2- 172.21.15.5:0/3976242540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070bf0 0x7f5b8413ce40 secure :-1 s=READY pgs=6591 cs=0 l=1 rev1=1 crypto rx=0x7f5b8409e3b0 tx=0x7f5b7c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34061302a57c7ee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 1 -- 172.21.15.5:0/3976242540 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c011830 con 0x7f5b84070bf0 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: dump: 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.129 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 1 -- 172.21.15.5:0/3976242540 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c011990 con 0x7f5b84070bf0 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 1 -- 172.21.15.5:0/3976242540 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c011d20 con 0x7f5b84070bf0 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b88ae6700 10 monclient: dump: 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 5 monclient: authenticate success, global_id 27960 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 4 set_mon_vals no callback set 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 10 monclient: shutdown 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3976242540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070bf0 msgr2=0x7f5b8413ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3976242540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070bf0 0x7f5b8413ce40 secure :-1 s=READY pgs=6591 cs=0 l=1 rev1=1 crypto rx=0x7f5b8409e3b0 tx=0x7f5b7c009b30 comp rx=0 tx=0).stop 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.116+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3976242540 shutdown_connections 2024-09-16T10:52:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3976242540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070bf0 0x7f5b8413ce40 unknown :-1 s=CLOSED pgs=6591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3976242540 >> v2:172.21.15.143:3300/0 conn(0x7f5b84070760 0x7f5b8413a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3976242540 >> 172.21.15.5:0/3976242540 conn(0x7f5b840a12a0 msgr2=0x7f5b840a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3976242540 shutdown_connections 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3976242540 wait complete. 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient: monmap: 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 librados: starting msgr at 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 librados: starting objecter 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 Processor -- start 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- start start 2024-09-16T10:52:01.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 librados: setting wanted keys 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 librados: calling monclient init 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient: init 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-a con 0x7f5b84070380 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-c con 0x7f5b84070760 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b84070bf0 0x7f5b841ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): picked mon.noname-b con 0x7f5b84070bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84139980 con 0x7f5b84070380 2024-09-16T10:52:01.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8413aa40 con 0x7f5b84070bf0 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b840717d0 con 0x7f5b84070760 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b8aaea700 10 monclient(hunting): authenticate will time out at 979.699724s 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b892e7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b89ae8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.120+0000 7f5b892e7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46646/0 (socket says 172.21.15.5:46646) 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 1 -- 172.21.15.5:0/3210559846 learned_addr learned my addr 172.21.15.5:0/3210559846 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38378/0 (socket says 172.21.15.5:38378) 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 10 monclient(hunting): get_auth_request con 0x7f5b84070760 auth_method 0 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): get_auth_request con 0x7f5b84070380 auth_method 0 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 1 -- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 msgr2=0x7f5b8412f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8a2e9700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b84070bf0 0x7f5b841ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8a2e9700 10 monclient(hunting): get_auth_request con 0x7f5b84070bf0 auth_method 0 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8a2e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8a2e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8a2e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient(hunting): handle_auth_done global_id 27963 payload 824 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 -- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 msgr2=0x7f5b8412f8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 -- 172.21.15.5:0/3210559846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b84070bf0 msgr2=0x7f5b841ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b84070bf0 0x7f5b841ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 monclient: found mon.noname-a 2024-09-16T10:52:01.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 -- 172.21.15.5:0/3210559846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0094b0 con 0x7f5b84070380 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b89ae8700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 secure :-1 s=READY pgs=6592 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800cbe0 tx=0x7f5b7800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 client.?.objecter ms_handle_connect 0x7f5b84070380 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7800ab00 con 0x7f5b84070380 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: dump: 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.143 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 5 monclient: authenticate success, global_id 27963 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7800ac60 con 0x7f5b84070380 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780070d0 con 0x7f5b84070380 2024-09-16T10:52:01.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 monclient: dump: 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _renew_subs 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b77fff700 4 set_mon_vals no callback set 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b841372e0 con 0x7f5b84070380 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 client.27963.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _renew_subs 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8413c720 con 0x7f5b84070380 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 1 librados: init done 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8413ac80 con 0x7f5b84070380 2024-09-16T10:52:01.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5b78007230 con 0x7f5b84070380 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 --2- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 0x7f5b68039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7324+0+0 (secure 0 0 0) 0x7f5b780480c0 con 0x7f5b84070380 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 10 client.27963.objecter ms_dispatch 0x7f5b84000fd0 osd_map(70..70 src has 1..70) v4 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 3 client.27963.objecter handle_osd_map got epochs [70,70] > 0 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 3 client.27963.objecter handle_osd_map decoding full epoch 70 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b767fc700 20 client.27963.objecter dump_active .. 0 homeless 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 1 --2- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 0x7f5b68039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.124+0000 7f5b892e7700 10 monclient: get_auth_request con 0x7f5b68036f60 auth_method 0 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.128+0000 7f5b892e7700 1 --2- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 0x7f5b68039420 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7f5b8413a930 tx=0x7f5b80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.128+0000 7f5b767fc700 10 client.27963.objecter ms_handle_connect 0x7f5b68036f60 2024-09-16T10:52:01.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.132+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5b78019860 con 0x7f5b84070380 2024-09-16T10:52:01.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.132+0000 7f5b767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.132+0000 7f5b767fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:01.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b8aaea700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b8aaea700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b8aaea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5b8413d300 con 0x7f5b84070380 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b767fc700 1 -- 172.21.15.5:0/3210559846 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f5b78018070 con 0x7f5b84070380 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:01.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.364+0000 7f5b767fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:01.369 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 10 librados: watch_flush enter 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 10 librados: watch_flush exit 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 20 client.27963.objecter shutdown clearing up homeless session... 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 10 client.27963.objecter successfully canceled tick 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 msgr2=0x7f5b68039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 0x7f5b68039420 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7f5b8413a930 tx=0x7f5b80008040 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 10 monclient: shutdown 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 msgr2=0x7f5b8412f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 secure :-1 s=READY pgs=6592 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800cbe0 tx=0x7f5b7800a060 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 shutdown_connections 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b84070bf0 0x7f5b841ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5b84070760 0x7f5b8412f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5b68036f60 0x7f5b68039420 unknown :-1 s=CLOSED pgs=3663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 --2- 172.21.15.5:0/3210559846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5b84070380 0x7f5b8412f380 unknown :-1 s=CLOSED pgs=6592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 >> 172.21.15.5:0/3210559846 conn(0x7f5b840a12a0 msgr2=0x7f5b8406f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 shutdown_connections 2024-09-16T10:52:01.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 -- 172.21.15.5:0/3210559846 wait complete. 2024-09-16T10:52:01.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.368+0000 7f5b8aaea700 1 librados: shutdown 2024-09-16T10:52:01.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:52:01.383 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:52:01.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:52:01.384 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:52:01.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.492+0000 7faad359e700 10 monclient: get_monmap_and_config 2024-09-16T10:52:01.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.492+0000 7faad359e700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.492+0000 7faad359e700 10 monclient: monmap: 2024-09-16T10:52:01.497 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.497 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.497 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:01.496119+0000 2024-09-16T10:52:01.497 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:01.496119+0000 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:01.498 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:01.499 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:01.499 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 Processor -- start 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- start start 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient: init 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faad409e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faad409d6a0 0x7faad4138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): picked mon.noname-a con 0x7faad409d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad413b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): picked mon.noname-b con 0x7faad409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad4070e40 con 0x7faad409e8e0 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad40a01d0 con 0x7faad409d6a0 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad409fef0 con 0x7faad409dfd0 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient(hunting): authenticate will time out at 980.073005s 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad259c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faad409e8e0 0x7faad409ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51420/0 (socket says 172.21.15.5:51420) 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad259c700 1 -- 172.21.15.5:0/2075518461 learned_addr learned my addr 172.21.15.5:0/2075518461 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 --2- 172.21.15.5:0/2075518461 >> v2:172.21.15.5:3300/0 conn(0x7faad409d6a0 0x7faad4138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): get_auth_request con 0x7faad409d6a0 auth_method 0 2024-09-16T10:52:01.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): get_auth_request con 0x7faad409dfd0 auth_method 0 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient(hunting): handle_auth_done global_id 27966 payload 274 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 -- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 msgr2=0x7faad413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 1 -- 172.21.15.5:0/2075518461 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525502609 0 0) 0x7faad4070e40 con 0x7faad409e8e0 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 -- 172.21.15.5:0/2075518461 >> v1:172.21.15.5:6789/0 conn(0x7faad409e8e0 legacy=0x7faad409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 monclient: found mon.noname-a 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 -- 172.21.15.5:0/2075518461 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad409f9f0 con 0x7faad409d6a0 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:52:01.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad1d9b700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409d6a0 0x7faad4138b80 secure :-1 s=READY pgs=6594 cs=0 l=1 rev1=1 crypto rx=0x7faac8009880 tx=0x7faac8009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e79742575b2608 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 1 -- 172.21.15.5:0/2075518461 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8011830 con 0x7faad409d6a0 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: dump: 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.521 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 5 monclient: authenticate success, global_id 27966 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 1 -- 172.21.15.5:0/2075518461 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac8011990 con 0x7faad409d6a0 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 4 set_mon_vals no callback set 2024-09-16T10:52:01.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 1 -- 172.21.15.5:0/2075518461 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8011d20 con 0x7faad409d6a0 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad159a700 10 monclient: dump: 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.523 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.524 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 10 monclient: shutdown 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409d6a0 msgr2=0x7faad4138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409d6a0 0x7faad4138b80 secure :-1 s=READY pgs=6594 cs=0 l=1 rev1=1 crypto rx=0x7faac8009880 tx=0x7faac8009b90 comp rx=0 tx=0).stop 2024-09-16T10:52:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- 172.21.15.5:0/2075518461 shutdown_connections 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad413b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 --2- 172.21.15.5:0/2075518461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409d6a0 0x7faad4138b80 unknown :-1 s=CLOSED pgs=6594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- 172.21.15.5:0/2075518461 >> 172.21.15.5:0/2075518461 conn(0x7faad40a45d0 msgr2=0x7faad40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- 172.21.15.5:0/2075518461 shutdown_connections 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.496+0000 7faad359e700 1 -- 172.21.15.5:0/2075518461 wait complete. 2024-09-16T10:52:01.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: monmap: 2024-09-16T10:52:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:01.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:01.528 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:01.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:01.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:01.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:01.529 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 librados: starting msgr at 2024-09-16T10:52:01.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 librados: starting objecter 2024-09-16T10:52:01.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 Processor -- start 2024-09-16T10:52:01.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- start start 2024-09-16T10:52:01.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 librados: setting wanted keys 2024-09-16T10:52:01.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 librados: calling monclient init 2024-09-16T10:52:01.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: init 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad409d6a0 0x7faad41ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): picked mon.noname-b con 0x7faad409d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): picked mon.noname-c con 0x7faad409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409e8e0 0x7faad4134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): picked mon.noname-a con 0x7faad409e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41233a0 con 0x7faad409e8e0 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad41235f0 con 0x7faad409d6a0 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad409b0d0 con 0x7faad409dfd0 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient(hunting): authenticate will time out at 980.077658s 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409e8e0 0x7faad4134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46668/0 (socket says 172.21.15.5:46668) 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 -- 172.21.15.5:0/4154165087 learned_addr learned my addr 172.21.15.5:0/4154165087 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): get_auth_request con 0x7faad409dfd0 auth_method 0 2024-09-16T10:52:01.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): get_auth_request con 0x7faad409e8e0 auth_method 0 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad409d6a0 0x7faad41ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 10 monclient(hunting): get_auth_request con 0x7faad409d6a0 auth_method 0 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient(hunting): handle_auth_done global_id 22181 payload 824 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad409d6a0 msgr2=0x7faad41ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad409d6a0 0x7faad41ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409e8e0 msgr2=0x7faad4134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409e8e0 0x7faad4134130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 monclient: found mon.noname-c 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80094b0 con 0x7faad409dfd0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad409d6a0 0x7faad41ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad1d9b700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 secure :-1 s=READY pgs=4497 cs=0 l=1 rev1=1 crypto rx=0x7faac8009850 tx=0x7faac8004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 client.?.objecter ms_handle_connect 0x7faad409dfd0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac80036c0 con 0x7faad409dfd0 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:01.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: dump: 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac8003820 con 0x7faad409dfd0 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 5 monclient: authenticate success, global_id 22181 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _renew_subs 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad406cbd0 con 0x7faad409dfd0 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8003b90 con 0x7faad409dfd0 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faac2ffd700 10 monclient: dump: 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.538 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad0d99700 4 set_mon_vals no callback set 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 client.22181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _renew_subs 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad4122d50 con 0x7faad409dfd0 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.500+0000 7faad359e700 1 librados: init done 2024-09-16T10:52:01.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad359e700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 0x7faad409b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad4123780 con 0x7faad409b3e0 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 0x7faad409b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient: get_auth_request con 0x7faad409b3e0 auth_method 0 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7faac8021070 con 0x7faad409dfd0 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 --2- 172.21.15.5:0/4154165087 >> v2:172.21.15.143:6812/1098205034 conn(0x7faab4036f60 0x7faab4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7324+0+0 (secure 0 0 0) 0x7faac8049ae0 con 0x7faad409dfd0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 client.22181.objecter ms_dispatch 0x7faad4000fd0 osd_map(70..70 src has 1..70) v4 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 3 client.22181.objecter handle_osd_map got epochs [70,70] > 0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 3 client.22181.objecter handle_osd_map decoding full epoch 70 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 10 monclient(hunting): handle_auth_done global_id 27975 payload 824 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faad40a3a10 con 0x7faad409b3e0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 20 client.22181.objecter dump_active .. 0 homeless 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad2d9d700 1 --2- 172.21.15.5:0/4154165087 >> v2:172.21.15.143:6812/1098205034 conn(0x7faab4036f60 0x7faab4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad2d9d700 10 monclient: get_auth_request con 0x7faab4036f60 auth_method 0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 client.22181.objecter ms_handle_connect 0x7faad409b3e0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 client.22181.objecter resend_mon_ops 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 0x7faad409b7c0 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 crypto rx=0x7faac40011e0 tx=0x7faac40018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4010510 con 0x7faad409b3e0 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 monclient: dump: 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.542 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faad2d9d700 1 --2- 172.21.15.5:0/4154165087 >> v2:172.21.15.143:6812/1098205034 conn(0x7faab4036f60 0x7faab4039420 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7faad40a03e0 tx=0x7faacc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 client.22181.objecter ms_handle_connect 0x7faab4036f60 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7faac4010670 con 0x7faad409b3e0 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 msgr2=0x7faad409b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 0x7faad409b7c0 secure :-1 s=READY pgs=6595 cs=0 l=1 rev1=1 crypto rx=0x7faac40011e0 tx=0x7faac40018e0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 msgr2=0x7faad409b7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.504+0000 7faac2ffd700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409b3e0 0x7faad409b7c0 unknown :-1 s=CLOSED pgs=6595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:52:01.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 0x7faad40a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad4067380 con 0x7faad409bd00 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 0x7faad40a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient: get_auth_request con 0x7faad409bd00 auth_method 0 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 10 monclient(hunting): handle_auth_done global_id 27978 payload 824 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 1 -- 172.21.15.5:0/4154165087 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faad409f3d0 con 0x7faad409bd00 2024-09-16T10:52:01.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 client.22181.objecter ms_handle_connect 0x7faad409bd00 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 client.22181.objecter resend_mon_ops 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad259c700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 0x7faad40a2020 secure :-1 s=READY pgs=6596 cs=0 l=1 rev1=1 crypto rx=0x7faac40011e0 tx=0x7faac4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4022070 con 0x7faad409bd00 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 monclient: dump: 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.547 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7faac401dd80 con 0x7faad409bd00 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 msgr2=0x7faad40a2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 0x7faad40a2020 secure :-1 s=READY pgs=6596 cs=0 l=1 rev1=1 crypto rx=0x7faac40011e0 tx=0x7faac4001080 comp rx=0 tx=0).stop 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 msgr2=0x7faad40a2020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faac2ffd700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faad409bd00 0x7faad40a2020 secure :-1 s=CLOSED pgs=6596 cs=0 l=1 rev1=1 crypto rx=0x7faac40011e0 tx=0x7faac4001080 comp rx=0 tx=0).stop 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.549 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.550 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.552 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.553 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.553 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.553 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.553 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.553 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:52:01.554 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.555 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.556 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.557 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.558 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.559 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:52:01.560 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.561 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:52:01.562 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:52:01.563 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 librados: watch_flush enter 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 librados: watch_flush exit 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 20 client.22181.objecter shutdown clearing up homeless session... 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 client.22181.objecter successfully canceled tick 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 >> v2:172.21.15.143:6812/1098205034 conn(0x7faab4036f60 msgr2=0x7faab4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 --2- 172.21.15.5:0/4154165087 >> v2:172.21.15.143:6812/1098205034 conn(0x7faab4036f60 0x7faab4039420 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7faad40a03e0 tx=0x7faacc00b040 comp rx=0 tx=0).stop 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 10 monclient: shutdown 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad2d9d700 1 -- 172.21.15.5:0/4154165087 reap_dead start 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 msgr2=0x7faad4133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 secure :-1 s=READY pgs=4497 cs=0 l=1 rev1=1 crypto rx=0x7faac8009850 tx=0x7faac8004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 shutdown_connections 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 --2- 172.21.15.5:0/4154165087 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faad409dfd0 0x7faad4133bf0 unknown :-1 s=CLOSED pgs=4497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 >> 172.21.15.5:0/4154165087 conn(0x7faad40a45d0 msgr2=0x7faad406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 shutdown_connections 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 -- 172.21.15.5:0/4154165087 wait complete. 2024-09-16T10:52:01.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.516+0000 7faad359e700 1 librados: shutdown 2024-09-16T10:52:01.567 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:52:01.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:52:01.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:52:01.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 10 monclient: get_monmap_and_config 2024-09-16T10:52:01.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 10 monclient: monmap: 2024-09-16T10:52:01.687 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:01.687277+0000 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:01.687277+0000 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:01.688 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 1 Processor -- start 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 1 -- start start 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.684+0000 7f6c8a70b700 10 monclient: init 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-b con 0x7f6c840a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6c8409f780 0x7f6c840a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-c con 0x7f6c8409f780 addr v2:172.21.15.143:3300/0 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c841359b0 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c840a3a50 con 0x7f6c841359b0 2024-09-16T10:52:01.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c840a3c40 con 0x7f6c8409f780 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840a4300 con 0x7f6c840a2bf0 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient(hunting): authenticate will time out at 980.264397s 2024-09-16T10:52:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46672/0 (socket says 172.21.15.5:46672) 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 -- 172.21.15.5:0/2875802859 learned_addr learned my addr 172.21.15.5:0/2875802859 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): get_auth_request con 0x7f6c840a2bf0 auth_method 0 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 1 --2- 172.21.15.5:0/2875802859 >> v2:172.21.15.143:3300/0 conn(0x7f6c8409f780 0x7f6c840a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): get_auth_request con 0x7f6c8409f780 auth_method 0 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient(hunting): handle_auth_done global_id 22184 payload 274 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 -- 172.21.15.5:0/2875802859 >> v2:172.21.15.143:3300/0 conn(0x7f6c8409f780 msgr2=0x7f6c840a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 --2- 172.21.15.5:0/2875802859 >> v2:172.21.15.143:3300/0 conn(0x7f6c8409f780 0x7f6c840a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 -- 172.21.15.5:0/2875802859 >> v1:172.21.15.143:6789/0 conn(0x7f6c841359b0 legacy=0x7f6c84135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 monclient: found mon.noname-b 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 -- 172.21.15.5:0/2875802859 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8409fda0 con 0x7f6c840a2bf0 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c89709700 1 --2- 172.21.15.5:0/2875802859 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 secure :-1 s=READY pgs=4498 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009910 tx=0x7f6c80009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53fe1755cd651d9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 1 -- 172.21.15.5:0/2875802859 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011830 con 0x7f6c840a2bf0 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: dump: 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.696 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 5 monclient: authenticate success, global_id 22184 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 1 -- 172.21.15.5:0/2875802859 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80011990 con 0x7f6c840a2bf0 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 4 set_mon_vals no callback set 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 1 -- 172.21.15.5:0/2875802859 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80011d40 con 0x7f6c840a2bf0 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c7bfff700 10 monclient: dump: 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: shutdown 2024-09-16T10:52:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- 172.21.15.5:0/2875802859 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 msgr2=0x7f6c840a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/2875802859 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 secure :-1 s=READY pgs=4498 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009910 tx=0x7f6c80009c20 comp rx=0 tx=0).stop 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- 172.21.15.5:0/2875802859 shutdown_connections 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/2875802859 >> v2:172.21.15.143:3300/0 conn(0x7f6c8409f780 0x7f6c840a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/2875802859 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c840a2fd0 unknown :-1 s=CLOSED pgs=4498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- 172.21.15.5:0/2875802859 >> 172.21.15.5:0/2875802859 conn(0x7f6c840a53e0 msgr2=0x7f6c84073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- 172.21.15.5:0/2875802859 shutdown_connections 2024-09-16T10:52:01.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 1 -- 172.21.15.5:0/2875802859 wait complete. 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.688+0000 7f6c8a70b700 10 monclient: monmap: 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:01.706 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 librados: starting msgr at 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 librados: starting objecter 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 Processor -- start 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- start start 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 librados: setting wanted keys 2024-09-16T10:52:01.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 librados: calling monclient init 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: init 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c8409f780 0x7f6c841ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-b con 0x7f6c8409f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c84134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-c con 0x7f6c840a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c841359b0 0x7f6c84135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): picked mon.noname-a con 0x7f6c841359b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c841ca8d0 con 0x7f6c841359b0 2024-09-16T10:52:01.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84070460 con 0x7f6c8409f780 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84122200 con 0x7f6c840a2bf0 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient(hunting): authenticate will time out at 980.269463s 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c84134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c841359b0 0x7f6c84135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c84134b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46688/0 (socket says 172.21.15.5:46688) 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c841359b0 0x7f6c84135040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38424/0 (socket says 172.21.15.5:38424) 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 1 -- 172.21.15.5:0/1143393671 learned_addr learned my addr 172.21.15.5:0/1143393671 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c8409f780 0x7f6c841ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): get_auth_request con 0x7f6c840a2bf0 auth_method 0 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): get_auth_request con 0x7f6c8409f780 auth_method 0 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): get_auth_request con 0x7f6c841359b0 auth_method 0 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89f0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient(hunting): handle_auth_done global_id 34180 payload 824 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 msgr2=0x7f6c84134b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c840a2bf0 0x7f6c84134b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c841359b0 msgr2=0x7f6c84135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6c841359b0 0x7f6c84135040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 monclient: found mon.noname-b 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80009570 con 0x7f6c8409f780 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c89709700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c8409f780 0x7f6c841ca390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c80016000 tx=0x7f6c80004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 client.?.objecter ms_handle_connect 0x7f6c8409f780 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:01.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80004510 con 0x7f6c8409f780 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: dump: 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.713 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 5 monclient: authenticate success, global_id 34180 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c800036c0 con 0x7f6c8409f780 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _renew_subs 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c841224e0 con 0x7f6c8409f780 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80003a70 con 0x7f6c8409f780 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c79ffb700 10 monclient: dump: 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 client.34180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:01.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _renew_subs 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c7b7fe700 4 set_mon_vals no callback set 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c840a6770 con 0x7f6c8409f780 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.692+0000 7f6c8a70b700 1 librados: init done 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c8a70b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 0x7f6c8411ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84121d00 con 0x7f6c8411e840 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 0x7f6c8411ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient: get_auth_request con 0x7f6c8411e840 auth_method 0 2024-09-16T10:52:01.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6c80003bd0 con 0x7f6c8409f780 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 10 monclient(hunting): handle_auth_done global_id 22190 payload 824 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 1 --2- 172.21.15.5:0/1143393671 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6c68036f60 0x7f6c68039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c840a4040 con 0x7f6c8411e840 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7324+0+0 (secure 0 0 0) 0x7f6c80021030 con 0x7f6c8409f780 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 10 client.34180.objecter ms_dispatch 0x7f6c84000fd0 osd_map(70..70 src has 1..70) v4 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 3 client.34180.objecter handle_osd_map got epochs [70,70] > 0 2024-09-16T10:52:01.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 3 client.34180.objecter handle_osd_map decoding full epoch 70 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c89f0a700 1 --2- 172.21.15.5:0/1143393671 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6c68036f60 0x7f6c68039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c88f08700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 0x7f6c8411ec20 secure :-1 s=READY pgs=4499 cs=0 l=1 rev1=1 crypto rx=0x7f6c840669e0 tx=0x7f6c74011f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 20 client.34180.objecter dump_active .. 0 homeless 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c89f0a700 10 monclient: get_auth_request con 0x7f6c68036f60 auth_method 0 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 10 client.34180.objecter ms_handle_connect 0x7f6c8411e840 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 10 client.34180.objecter resend_mon_ops 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c79ffb700 10 client.34180.objecter ms_handle_connect 0x7f6c68036f60 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.696+0000 7f6c89f0a700 1 --2- 172.21.15.5:0/1143393671 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6c68036f60 0x7f6c68039420 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7f6c84120eb0 tx=0x7f6c7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740043a0 con 0x7f6c8411e840 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:01.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 10 monclient: dump: 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.719 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f6c74004500 con 0x7f6c8411e840 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 msgr2=0x7f6c8411ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 0x7f6c8411ec20 secure :-1 s=READY pgs=4499 cs=0 l=1 rev1=1 crypto rx=0x7f6c840669e0 tx=0x7f6c74011f60 comp rx=0 tx=0).stop 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 msgr2=0x7f6c8411ec20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.700+0000 7f6c79ffb700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411e840 0x7f6c8411ec20 secure :-1 s=CLOSED pgs=4499 cs=0 l=1 rev1=1 crypto rx=0x7f6c840669e0 tx=0x7f6c74011f60 comp rx=0 tx=0).stop 2024-09-16T10:52:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c8a70b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:52:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 0x7f6c8406c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c8a70b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84074d50 con 0x7f6c8411f160 2024-09-16T10:52:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c88f08700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 0x7f6c8406c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c88f08700 10 monclient: get_auth_request con 0x7f6c8411f160 auth_method 0 2024-09-16T10:52:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c88f08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c88f08700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.724+0000 7f6c88f08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 10 monclient(hunting): handle_auth_done global_id 22193 payload 824 2024-09-16T10:52:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 1 -- 172.21.15.5:0/1143393671 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c840a2fd0 con 0x7f6c8411f160 2024-09-16T10:52:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c88f08700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 0x7f6c8406c320 secure :-1 s=READY pgs=4500 cs=0 l=1 rev1=1 crypto rx=0x7f6c7401b000 tx=0x7f6c74004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 client.34180.objecter ms_handle_connect 0x7f6c8411f160 2024-09-16T10:52:01.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 client.34180.objecter resend_mon_ops 2024-09-16T10:52:01.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400fee0 con 0x7f6c8411f160 2024-09-16T10:52:01.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:01.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 monclient: dump: 2024-09-16T10:52:01.735 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.736 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f6c740283a0 con 0x7f6c8411f160 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:52:01.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:01.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 msgr2=0x7f6c8406c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 0x7f6c8406c320 secure :-1 s=READY pgs=4500 cs=0 l=1 rev1=1 crypto rx=0x7f6c7401b000 tx=0x7f6c74004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 msgr2=0x7f6c8406c320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c79ffb700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6c8411f160 0x7f6c8406c320 secure :-1 s=CLOSED pgs=4500 cs=0 l=1 rev1=1 crypto rx=0x7f6c7401b000 tx=0x7f6c74004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.739 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:52:01.739 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:01.739 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:01.739 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:52:01.740 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:52:01.740 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:52:01.740 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:52:01.740 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:52:01.740 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:52:01.741 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.742 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.742 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.742 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.742 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.742 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.743 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.743 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.743 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.743 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.743 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.744 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.745 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.745 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.745 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.745 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.745 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.746 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.746 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:52:01.746 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.747 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.748 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:52:01.748 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:52:01.748 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:52:01.748 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:52:01.748 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.749 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.750 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.750 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.750 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.750 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.750 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:52:01.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.752 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:52:01.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.752 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:01.752 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:01.752 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.753 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.753 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.753 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:52:01.753 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.754 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.755 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.755 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.755 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.755 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.755 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.756 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.756 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.756 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:01.757 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:01.757 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.760 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.761 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.761 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:52:01.761 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.762 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.763 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.763 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.763 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.763 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:01.764 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.765 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.765 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.765 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:52:01.765 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.766 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.767 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.767 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.767 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.767 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.768 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.768 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.768 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.769 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:01.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.770 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.770 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:52:01.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.770 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:52:01.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.772 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.772 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.772 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-16T10:52:01.772 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.772 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.773 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:52:01.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.773 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.773 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.773 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:01.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.774 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.774 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:52:01.774 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:52:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 10 librados: watch_flush enter 2024-09-16T10:52:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 10 librados: watch_flush exit 2024-09-16T10:52:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 20 client.34180.objecter shutdown clearing up homeless session... 2024-09-16T10:52:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 10 client.34180.objecter successfully canceled tick 2024-09-16T10:52:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6c68036f60 msgr2=0x7f6c68039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/1143393671 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6c68036f60 0x7f6c68039420 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7f6c84120eb0 tx=0x7f6c7c009040 comp rx=0 tx=0).stop 2024-09-16T10:52:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 10 monclient: shutdown 2024-09-16T10:52:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c8409f780 msgr2=0x7f6c841ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 --2- 172.21.15.5:0/1143393671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c8409f780 0x7f6c841ca390 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c80016000 tx=0x7f6c80004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c89f0a700 1 -- 172.21.15.5:0/1143393671 reap_dead start 2024-09-16T10:52:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c89f0a700 1 -- 172.21.15.5:0/1143393671 reap_dead start 2024-09-16T10:52:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 shutdown_connections 2024-09-16T10:52:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 >> 172.21.15.5:0/1143393671 conn(0x7f6c840a53e0 msgr2=0x7f6c840709b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 shutdown_connections 2024-09-16T10:52:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 -- 172.21.15.5:0/1143393671 wait complete. 2024-09-16T10:52:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.728+0000 7f6c8a70b700 1 librados: shutdown 2024-09-16T10:52:01.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:52:01.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:52:01.782 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:52:01.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.888+0000 7f9e48e32700 10 monclient: get_monmap_and_config 2024-09-16T10:52:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.888+0000 7f9e48e32700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.888+0000 7f9e48e32700 10 monclient: monmap: 2024-09-16T10:52:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:01.892418+0000 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:01.892418+0000 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:01.894 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 Processor -- start 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 -- start start 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient: init 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 0x7f9e440a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-c con 0x7f9e440a2c80 addr v2:172.21.15.143:3300/0 2024-09-16T10:52:01.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-b con 0x7f9e440a35a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9e44135f80 0x7f9e441363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-a con 0x7f9e44135f80 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e441368f0 con 0x7f9e44135f80 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e440a3db0 con 0x7f9e440a2c80 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e440a4b70 con 0x7f9e440a35a0 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient(hunting): authenticate will time out at 980.469690s 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9e44135f80 0x7f9e441363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46708/0 (socket says 172.21.15.5:46708) 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 1 -- 172.21.15.5:0/3245092828 learned_addr learned my addr 172.21.15.5:0/3245092828 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 0x7f9e440a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): get_auth_request con 0x7f9e440a35a0 auth_method 0 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): get_auth_request con 0x7f9e44135f80 auth_method 0 2024-09-16T10:52:01.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): get_auth_request con 0x7f9e440a2c80 auth_method 0 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient(hunting): handle_auth_done global_id 27990 payload 274 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 -- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 msgr2=0x7f9e44135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 -- 172.21.15.5:0/3245092828 >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 msgr2=0x7f9e440a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 --2- 172.21.15.5:0/3245092828 >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 0x7f9e440a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 monclient: found mon.noname-a 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 -- 172.21.15.5:0/3245092828 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e440a0850 con 0x7f9e44135f80 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e437fe700 1 --2- 172.21.15.5:0/3245092828 >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 0x7f9e440a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e42ffd700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e43fff700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e441363b0 secure :-1 s=READY pgs=6598 cs=0 l=1 rev1=1 crypto rx=0x7f9e440a81c0 tx=0x7f9e3c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6094e1c11a90b01a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 1 -- 172.21.15.5:0/3245092828 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c011830 con 0x7f9e44135f80 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: dump: 2024-09-16T10:52:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 5 monclient: authenticate success, global_id 27990 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 1 -- 172.21.15.5:0/3245092828 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c011990 con 0x7f9e44135f80 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e48e32700 4 set_mon_vals no callback set 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 1 -- 172.21.15.5:0/3245092828 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c011d20 con 0x7f9e44135f80 2024-09-16T10:52:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.892+0000 7f9e427fc700 10 monclient: dump: 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 10 monclient: shutdown 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 msgr2=0x7f9e441363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e441363b0 secure :-1 s=READY pgs=6598 cs=0 l=1 rev1=1 crypto rx=0x7f9e440a81c0 tx=0x7f9e3c009b30 comp rx=0 tx=0).stop 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- 172.21.15.5:0/3245092828 shutdown_connections 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e441363b0 unknown :-1 s=CLOSED pgs=6598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 --2- 172.21.15.5:0/3245092828 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e44135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 --2- 172.21.15.5:0/3245092828 >> v2:172.21.15.143:3300/0 conn(0x7f9e440a2c80 0x7f9e440a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- 172.21.15.5:0/3245092828 >> 172.21.15.5:0/3245092828 conn(0x7f9e440a4d80 msgr2=0x7f9e440a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- 172.21.15.5:0/3245092828 shutdown_connections 2024-09-16T10:52:01.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- 172.21.15.5:0/3245092828 wait complete. 2024-09-16T10:52:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 10 monclient: build_initial_monmap 2024-09-16T10:52:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 10 monclient: monmap: 2024-09-16T10:52:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 librados: starting msgr at 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 librados: starting objecter 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 Processor -- start 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 -- start start 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 librados: setting wanted keys 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 1 librados: calling monclient init 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 10 monclient: init 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.896+0000 7f9e48e32700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:52:01.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e440a2c80 0x7f9e441caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-b con 0x7f9e440a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e441cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-c con 0x7f9e440a35a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): picked mon.noname-a con 0x7f9e44135f80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44125160 con 0x7f9e44135f80 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44123380 con 0x7f9e440a2c80 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4409e850 con 0x7f9e440a35a0 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): authenticate will time out at 980.476210s 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38440/0 (socket says 172.21.15.5:38440) 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e441cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 -- 172.21.15.5:0/735558030 learned_addr learned my addr 172.21.15.5:0/735558030 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): get_auth_request con 0x7f9e44135f80 auth_method 0 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): get_auth_request con 0x7f9e440a35a0 auth_method 0 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e440a2c80 0x7f9e441caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): get_auth_request con 0x7f9e440a2c80 auth_method 0 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient(hunting): handle_auth_done global_id 27993 payload 824 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 msgr2=0x7f9e441cafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9e440a35a0 0x7f9e441cafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e440a2c80 msgr2=0x7f9e441caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e440a2c80 0x7f9e441caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient: _finish_hunting 0 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 monclient: found mon.noname-a 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e340094b0 con 0x7f9e44135f80 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e440a2c80 0x7f9e441caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e43fff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 secure :-1 s=READY pgs=6599 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009b70 tx=0x7f9e3c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 client.?.objecter ms_handle_connect 0x7f9e44135f80 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c004300 con 0x7f9e44135f80 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: dump: 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: _finish_auth 0 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:52:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 5 monclient: authenticate success, global_id 27993 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient: _renew_subs 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3c01f070 con 0x7f9e44135f80 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c009b30 con 0x7f9e44135f80 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3c0036c0 con 0x7f9e44135f80 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 10 monclient: dump: 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e41ffb700 4 set_mon_vals no callback set 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 client.27993.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient: _renew_subs 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c0094b0 con 0x7f9e44135f80 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 librados: init done 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 0x7f9e4402b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44124f30 con 0x7f9e44121e70 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 0x7f9e4402b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient: get_auth_request con 0x7f9e44121e70 auth_method 0 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 10 monclient(hunting): handle_auth_done global_id 34195 payload 824 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e441252d0 con 0x7f9e44121e70 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9e3c021030 con 0x7f9e44135f80 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.900+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 0x7f9e4402b6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e34011890 tx=0x7f9e34011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 --2- 172.21.15.5:0/735558030 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9e24036f60 0x7f9e24039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 client.27993.objecter ms_handle_connect 0x7f9e44121e70 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 client.27993.objecter resend_mon_ops 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7300+0+0 (secure 0 0 0) 0x7f9e3c049890 con 0x7f9e44135f80 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 client.27993.objecter ms_dispatch 0x7f9e44000fd0 osd_map(71..71 src has 1..71) v4 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 3 client.27993.objecter handle_osd_map got epochs [71,71] > 0 2024-09-16T10:52:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 3 client.27993.objecter handle_osd_map decoding full epoch 71 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 20 client.27993.objecter dump_active .. 0 homeless 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e42ffd700 1 --2- 172.21.15.5:0/735558030 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9e24036f60 0x7f9e24039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400fa60 con 0x7f9e44121e70 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 monclient: dump: 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e42ffd700 10 monclient: get_auth_request con 0x7f9e24036f60 auth_method 0 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f9e34011e90 con 0x7f9e44121e70 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 msgr2=0x7f9e4402b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 0x7f9e4402b6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e34011890 tx=0x7f9e34011c50 comp rx=0 tx=0).stop 2024-09-16T10:52:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 msgr2=0x7f9e4402b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44121e70 0x7f9e4402b6a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e34011890 tx=0x7f9e34011c50 comp rx=0 tx=0).stop 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e3bfff700 10 client.27993.objecter ms_handle_connect 0x7f9e24036f60 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.904+0000 7f9e42ffd700 1 --2- 172.21.15.5:0/735558030 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9e24036f60 0x7f9e24039420 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f9e44123960 tx=0x7f9e2c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.912+0000 7f9e48e32700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.912+0000 7f9e48e32700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 0x7f9e4402bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.912+0000 7f9e48e32700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.912+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44067380 con 0x7f9e44122250 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 0x7f9e4402bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient: get_auth_request con 0x7f9e44122250 auth_method 0 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 10 monclient(hunting): handle_auth_done global_id 34198 payload 824 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 1 -- 172.21.15.5:0/735558030 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9e44135c70 con 0x7f9e44122250 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e437fe700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 0x7f9e4402bbe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e34011890 tx=0x7f9e34004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 client.27993.objecter ms_handle_connect 0x7f9e44122250 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 client.27993.objecter resend_mon_ops 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34006580 con 0x7f9e44122250 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:52:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 monclient: dump: 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:01.936 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f9e340066e0 con 0x7f9e44122250 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 msgr2=0x7f9e4402bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 0x7f9e4402bbe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9e34011890 tx=0x7f9e34004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 msgr2=0x7f9e4402bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:01.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e3bfff700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e44122250 0x7f9e4402bbe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 18, 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:52:01.938 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 5, 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.939 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.940 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.941 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:01.942 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:52:01.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.945 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.946 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.947 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:01.948 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:52:01.949 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.950 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:01.951 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:52:01.952 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:52:01.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 10 librados: watch_flush enter 2024-09-16T10:52:01.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 10 librados: watch_flush exit 2024-09-16T10:52:01.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 20 client.27993.objecter shutdown clearing up homeless session... 2024-09-16T10:52:01.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 10 client.27993.objecter successfully canceled tick 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9e24036f60 msgr2=0x7f9e24039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 --2- 172.21.15.5:0/735558030 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9e24036f60 0x7f9e24039420 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f9e44123960 tx=0x7f9e2c00b040 comp rx=0 tx=0).stop 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 10 monclient: shutdown 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e43fff700 1 -- 172.21.15.5:0/735558030 reap_dead start 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 msgr2=0x7f9e44121160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 secure :-1 s=READY pgs=6599 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009b70 tx=0x7f9e3c004770 comp rx=0 tx=0).stop 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 shutdown_connections 2024-09-16T10:52:01.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 --2- 172.21.15.5:0/735558030 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e44135f80 0x7f9e44121160 unknown :-1 s=CLOSED pgs=6599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:01.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 >> 172.21.15.5:0/735558030 conn(0x7f9e440a4d80 msgr2=0x7f9e441352b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:01.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 shutdown_connections 2024-09-16T10:52:01.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 -- 172.21.15.5:0/735558030 wait complete. 2024-09-16T10:52:01.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:01.916+0000 7f9e48e32700 1 librados: shutdown 2024-09-16T10:52:01.956 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:52:01.957 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T10:52:01.957 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:52:01.957 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T10:52:01.957 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:52:01.957 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T10:52:01.957 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:52:01.998 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T10:52:01.998 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:52:01.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.108+0000 7f85ff59e700 10 monclient: get_monmap_and_config 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.108+0000 7f85ff59e700 10 monclient: build_initial_monmap 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.108+0000 7f85ff59e700 10 monclient: monmap: 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:02.111492+0000 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:02.111492+0000 2024-09-16T10:52:02.112 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:02.113 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.108+0000 7f85ff59e700 1 Processor -- start 2024-09-16T10:52:02.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- start start 2024-09-16T10:52:02.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: init 2024-09-16T10:52:02.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:02.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f86000a2bf0 0x7f86000a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-a con 0x7f86000a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-b con 0x7f860009f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 0x7f8600135dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-c con 0x7f86001359b0 addr v2:172.21.15.143:3300/0 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000a3a50 con 0x7f86000a2bf0 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000a3c40 con 0x7f86001359b0 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000a4300 con 0x7f860009f780 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient(hunting): authenticate will time out at 980.688528s 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:46720/0 (socket says 172.21.15.5:46720) 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 -- 172.21.15.5:0/2974838948 learned_addr learned my addr 172.21.15.5:0/2974838948 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): get_auth_request con 0x7f860009f780 auth_method 0 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:02.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fed9d700 1 -- 172.21.15.5:0/2974838948 >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 msgr2=0x7f8600135dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.5:3300/0 conn(0x7f86000a2bf0 0x7f86000a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fed9d700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 0x7f8600135dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): get_auth_request con 0x7f86000a2bf0 auth_method 0 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:02.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient(hunting): handle_auth_done global_id 22208 payload 274 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 -- 172.21.15.5:0/2974838948 >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 msgr2=0x7f8600135dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 0x7f8600135dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 -- 172.21.15.5:0/2974838948 >> v2:172.21.15.5:3300/0 conn(0x7f86000a2bf0 msgr2=0x7f86000a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.5:3300/0 conn(0x7f86000a2bf0 0x7f86000a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient: _finish_hunting 0 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 monclient: found mon.noname-b 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 -- 172.21.15.5:0/2974838948 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860009fda0 con 0x7f860009f780 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85f7fff700 1 --2- 172.21.15.5:0/2974838948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 secure :-1 s=READY pgs=4503 cs=0 l=1 rev1=1 crypto rx=0x7f860009fbb0 tx=0x7f85e8009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f187e9865dcf2b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 1 -- 172.21.15.5:0/2974838948 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8011830 con 0x7f860009f780 2024-09-16T10:52:02.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: dump: 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: _finish_auth 0 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: _check_auth_tickets 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 5 monclient: authenticate success, global_id 22208 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 1 -- 172.21.15.5:0/2974838948 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e8011990 con 0x7f860009f780 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 4 set_mon_vals no callback set 2024-09-16T10:52:02.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 1 -- 172.21.15.5:0/2974838948 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8011d20 con 0x7f860009f780 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85fdd9b700 10 monclient: dump: 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:02.127 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: shutdown 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- 172.21.15.5:0/2974838948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 msgr2=0x7f86000a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- 172.21.15.5:0/2974838948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 secure :-1 s=READY pgs=4503 cs=0 l=1 rev1=1 crypto rx=0x7f860009fbb0 tx=0x7f85e8009bf0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- 172.21.15.5:0/2974838948 shutdown_connections 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.143:3300/0 conn(0x7f86001359b0 0x7f8600135dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- 172.21.15.5:0/2974838948 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f860009f780 0x7f86000a3510 unknown :-1 s=CLOSED pgs=4503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 --2- 172.21.15.5:0/2974838948 >> v2:172.21.15.5:3300/0 conn(0x7f86000a2bf0 0x7f86000a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- 172.21.15.5:0/2974838948 >> 172.21.15.5:0/2974838948 conn(0x7f86000a53e0 msgr2=0x7f8600073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- 172.21.15.5:0/2974838948 shutdown_connections 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 1 -- 172.21.15.5:0/2974838948 wait complete. 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: build_initial_monmap 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.112+0000 7f85ff59e700 10 monclient: monmap: 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:02.129 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 librados: starting msgr at 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 librados: starting objecter 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 Processor -- start 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- start start 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 librados: setting wanted keys 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 librados: calling monclient init 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: init 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 0x7f86001caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-b con 0x7f860009f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-a con 0x7f86000a2bf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 0x7f8600122140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): picked mon.noname-c con 0x7f86001359b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600125730 con 0x7f86000a2bf0 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000a6900 con 0x7f860009f780 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8600070610 con 0x7f86001359b0 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:02.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient(hunting): authenticate will time out at 980.693060s 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fe59c700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 msgr2=0x7f86001caf30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fe59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 0x7f86001caf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 0x7f8600122140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38452/0 (socket says 172.21.15.5:38452) 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 1 -- 172.21.15.5:0/3109113661 learned_addr learned my addr 172.21.15.5:0/3109113661 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): get_auth_request con 0x7f86001359b0 auth_method 0 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): get_auth_request con 0x7f86000a2bf0 auth_method 0 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient(hunting): handle_auth_done global_id 22211 payload 824 2024-09-16T10:52:02.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 -- 172.21.15.5:0/3109113661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 msgr2=0x7f86001caf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 0x7f86001caf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 -- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 msgr2=0x7f86001cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient: _finish_hunting 0 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 monclient: found mon.noname-c 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 -- 172.21.15.5:0/3109113661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8009570 con 0x7f86001359b0 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f7fff700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 client.?.objecter ms_handle_connect 0x7f86001359b0 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fed9d700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 0x7f8600122140 secure :-1 s=READY pgs=4504 cs=0 l=1 rev1=1 crypto rx=0x7f85f800bd80 tx=0x7f85f8009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:02.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800cbb0 con 0x7f86001359b0 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: dump: 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.135 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: _finish_auth 0 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8006e90 con 0x7f86001359b0 2024-09-16T10:52:02.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 5 monclient: authenticate success, global_id 22211 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _renew_subs 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860009f4e0 con 0x7f86001359b0 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8015470 con 0x7f86001359b0 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85f6ffd700 10 monclient: dump: 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.137 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85fd59a700 4 set_mon_vals no callback set 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 client.22211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _renew_subs 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8600123c00 con 0x7f86001359b0 2024-09-16T10:52:02.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.116+0000 7f85ff59e700 1 librados: init done 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85ff59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85ff59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85ff59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86000a2fd0 con 0x7f86001359b0 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f85f8012070 con 0x7f86001359b0 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 1 --2- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 0x7f85e00393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7300+0+0 (secure 0 0 0) 0x7f85f8048370 con 0x7f86001359b0 2024-09-16T10:52:02.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 10 client.22211.objecter ms_dispatch 0x7f8600000fd0 osd_map(71..71 src has 1..71) v4 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 3 client.22211.objecter handle_osd_map got epochs [71,71] > 0 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 3 client.22211.objecter handle_osd_map decoding full epoch 71 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 20 client.22211.objecter dump_active .. 0 homeless 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85fe59c700 1 --2- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 0x7f85e00393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85fe59c700 10 monclient: get_auth_request con 0x7f85e0036f10 auth_method 0 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85f6ffd700 10 client.22211.objecter ms_handle_connect 0x7f85e0036f10 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.120+0000 7f85fe59c700 1 --2- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 0x7f85e00393d0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009550 tx=0x7f85f0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.128+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f85f80155d0 con 0x7f86001359b0 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.128+0000 7f85f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.128+0000 7f85f6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:02.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85ff59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:52:02.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85ff59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:02.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85ff59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:02.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f86000008d0 con 0x7f86001359b0 2024-09-16T10:52:02.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85f6ffd700 1 -- 172.21.15.5:0/3109113661 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f85f80143d0 con 0x7f86001359b0 2024-09-16T10:52:02.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:02.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.364+0000 7f85f6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:02.368 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 10 librados: watch_flush enter 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 10 librados: watch_flush exit 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 20 client.22211.objecter shutdown clearing up homeless session... 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 10 client.22211.objecter successfully canceled tick 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 msgr2=0x7f85e00393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 0x7f85e00393d0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009550 tx=0x7f85f0009240 comp rx=0 tx=0).stop 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 10 monclient: shutdown 2024-09-16T10:52:02.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 msgr2=0x7f8600122140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 0x7f8600122140 secure :-1 s=READY pgs=4504 cs=0 l=1 rev1=1 crypto rx=0x7f85f800bd80 tx=0x7f85f8009ef0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 shutdown_connections 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f86001359b0 0x7f8600122140 unknown :-1 s=CLOSED pgs=4504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> v2:172.21.15.143:6812/1098205034 conn(0x7f85e0036f10 0x7f85e00393d0 unknown :-1 s=CLOSED pgs=3668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f86000a2bf0 0x7f86001cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 --2- 172.21.15.5:0/3109113661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860009f780 0x7f86001caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 >> 172.21.15.5:0/3109113661 conn(0x7f86000a53e0 msgr2=0x7f86000a6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 shutdown_connections 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 -- 172.21.15.5:0/3109113661 wait complete. 2024-09-16T10:52:02.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:02.368+0000 7f85ff59e700 1 librados: shutdown 2024-09-16T10:52:05.383 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.500+0000 7f52a17b4700 10 monclient: get_monmap_and_config 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.500+0000 7f52a17b4700 10 monclient: build_initial_monmap 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.500+0000 7f52a17b4700 10 monclient: monmap: 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:05.503713+0000 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:05.503713+0000 2024-09-16T10:52:05.504 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:05.505 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.500+0000 7f52a17b4700 1 Processor -- start 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- start start 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient: init 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f529c0a2c80 0x7f529c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-a con 0x7f529c0a2c80 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f529c0a35a0 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f529c136150 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:05.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0a3db0 con 0x7f529c0a35a0 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529c0a4b70 con 0x7f529c136150 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529c0a0850 con 0x7f529c0a2c80 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient(hunting): authenticate will time out at 984.080738s 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f529c0a2c80 0x7f529c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529b7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f529c0a35a0 0x7f529c135a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57224/0 (socket says 172.21.15.5:57224) 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a0fb3700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f529c136150 legacy=0x7f529c13dca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529b7fe700 1 -- 172.21.15.5:0/2146144188 learned_addr learned my addr 172.21.15.5:0/2146144188 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): get_auth_request con 0x7f529c0a2c80 auth_method 0 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:05.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:05.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient(hunting): handle_auth_done global_id 28002 payload 274 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 -- 172.21.15.5:0/2146144188 >> v1:172.21.15.143:6789/0 conn(0x7f529c136150 legacy=0x7f529c13dca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 -- 172.21.15.5:0/2146144188 >> v1:172.21.15.5:6789/0 conn(0x7f529c0a35a0 legacy=0x7f529c135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 monclient: found mon.noname-a 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 -- 172.21.15.5:0/2146144188 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c0a4180 con 0x7f529c0a2c80 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529bfff700 1 --2- 172.21.15.5:0/2146144188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a2c80 0x7f529c0a3060 secure :-1 s=READY pgs=6603 cs=0 l=1 rev1=1 crypto rx=0x7f528c009850 tx=0x7f528c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2f53315d44568 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 1 -- 172.21.15.5:0/2146144188 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c004230 con 0x7f529c0a2c80 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:05.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: dump: 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: _finish_auth 0 2024-09-16T10:52:05.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: _check_auth_tickets 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 1 -- 172.21.15.5:0/2146144188 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c004390 con 0x7f529c0a2c80 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 5 monclient: authenticate success, global_id 28002 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 1 -- 172.21.15.5:0/2146144188 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0036c0 con 0x7f529c0a2c80 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 4 set_mon_vals no callback set 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f529affd700 10 monclient: dump: 2024-09-16T10:52:05.513 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:05.514 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 10 monclient: shutdown 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- 172.21.15.5:0/2146144188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a2c80 msgr2=0x7f529c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 --2- 172.21.15.5:0/2146144188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a2c80 0x7f529c0a3060 secure :-1 s=READY pgs=6603 cs=0 l=1 rev1=1 crypto rx=0x7f528c009850 tx=0x7f528c009b60 comp rx=0 tx=0).stop 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- 172.21.15.5:0/2146144188 shutdown_connections 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 --2- 172.21.15.5:0/2146144188 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a2c80 0x7f529c0a3060 unknown :-1 s=CLOSED pgs=6603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- 172.21.15.5:0/2146144188 >> 172.21.15.5:0/2146144188 conn(0x7f529c0a4d80 msgr2=0x7f529c0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:05.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- 172.21.15.5:0/2146144188 shutdown_connections 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.504+0000 7f52a17b4700 1 -- 172.21.15.5:0/2146144188 wait complete. 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient: build_initial_monmap 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient: monmap: 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:05.522 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 librados: starting msgr at 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 librados: starting objecter 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 Processor -- start 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 -- start start 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 librados: setting wanted keys 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 librados: calling monclient init 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient: init 2024-09-16T10:52:05.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-c con 0x7f529c0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-a con 0x7f529c0a35a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 0x7f529c1c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): picked mon.noname-b con 0x7f529c136150 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:05.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c126160 con 0x7f529c0a35a0 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c0a7660 con 0x7f529c136150 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c123eb0 con 0x7f529c0a2c80 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a17b4700 10 monclient(hunting): authenticate will time out at 984.087069s 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a0fb3700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 msgr2=0x7f529c1c7370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f52a0fb3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 0x7f529c1c7370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:41784/0 (socket says 172.21.15.5:41784) 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 1 -- 172.21.15.5:0/728167521 learned_addr learned my addr 172.21.15.5:0/728167521 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:05.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40554/0 (socket says 172.21.15.5:40554) 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): get_auth_request con 0x7f529c0a2c80 auth_method 0 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): get_auth_request con 0x7f529c0a35a0 auth_method 0 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:05.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 10 monclient(hunting): handle_auth_done global_id 22214 payload 824 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.508+0000 7f529bfff700 1 -- 172.21.15.5:0/728167521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 msgr2=0x7f529c1c7370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 0x7f529c1c7370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 -- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 msgr2=0x7f529c135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 monclient: found mon.noname-c 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:05.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 -- 172.21.15.5:0/728167521 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0094b0 con 0x7f529c0a2c80 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529b7fe700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529bfff700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 secure :-1 s=READY pgs=4506 cs=0 l=1 rev1=1 crypto rx=0x7f528c000c00 tx=0x7f528c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 client.?.objecter ms_handle_connect 0x7f529c0a2c80 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c006ea0 con 0x7f529c0a2c80 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: dump: 2024-09-16T10:52:05.528 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: _finish_auth 0 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:05.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c011830 con 0x7f529c0a2c80 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 5 monclient: authenticate success, global_id 22214 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _renew_subs 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c135a20 con 0x7f529c0a2c80 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c011bc0 con 0x7f529c0a2c80 2024-09-16T10:52:05.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 monclient: dump: 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:05.531 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529a7fc700 4 set_mon_vals no callback set 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 client.22214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _renew_subs 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c126a00 con 0x7f529c0a2c80 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 1 librados: init done 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:05.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f528c021070 con 0x7f529c0a2c80 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c09e630 con 0x7f529c0a2c80 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 --2- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 0x7f52880393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7300+0+0 (secure 0 0 0) 0x7f528c049d90 con 0x7f529c0a2c80 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 client.22214.objecter ms_dispatch 0x7f529c000fd0 osd_map(71..71 src has 1..71) v4 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 3 client.22214.objecter handle_osd_map got epochs [71,71] > 0 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 3 client.22214.objecter handle_osd_map decoding full epoch 71 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 20 client.22214.objecter dump_active .. 0 homeless 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529b7fe700 1 --2- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 0x7f52880393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529b7fe700 10 monclient: get_auth_request con 0x7f5288036f10 auth_method 0 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f5298ff9700 10 client.22214.objecter ms_handle_connect 0x7f5288036f10 2024-09-16T10:52:05.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.512+0000 7f529b7fe700 1 --2- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 0x7f52880393d0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f529c126090 tx=0x7f5290007f30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:05.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.520+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f528c00bbd0 con 0x7f529c0a2c80 2024-09-16T10:52:05.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.520+0000 7f5298ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:05.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.520+0000 7f5298ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f52a17b4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f52a17b4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f52a17b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f529c09f0d0 con 0x7f529c0a2c80 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f5298ff9700 1 -- 172.21.15.5:0/728167521 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f528c029020 con 0x7f529c0a2c80 2024-09-16T10:52:05.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f5298ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:05.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.760+0000 7f5298ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:05.764 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 10 librados: watch_flush enter 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 10 librados: watch_flush exit 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 20 client.22214.objecter shutdown clearing up homeless session... 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 10 client.22214.objecter successfully canceled tick 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 msgr2=0x7f52880393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:05.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 0x7f52880393d0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f529c126090 tx=0x7f5290007f30 comp rx=0 tx=0).stop 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 10 monclient: shutdown 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 msgr2=0x7f529c1ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 secure :-1 s=READY pgs=4506 cs=0 l=1 rev1=1 crypto rx=0x7f528c000c00 tx=0x7f528c004770 comp rx=0 tx=0).stop 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 shutdown_connections 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f529c136150 0x7f529c1c7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5288036f10 0x7f52880393d0 unknown :-1 s=CLOSED pgs=3669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f529c0a35a0 0x7f529c135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 --2- 172.21.15.5:0/728167521 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f529c0a2c80 0x7f529c1ca870 unknown :-1 s=CLOSED pgs=4506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:05.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 >> 172.21.15.5:0/728167521 conn(0x7f529c0a4d80 msgr2=0x7f529c0a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:05.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 shutdown_connections 2024-09-16T10:52:05.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 -- 172.21.15.5:0/728167521 wait complete. 2024-09-16T10:52:05.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:05.764+0000 7f52a17b4700 1 librados: shutdown 2024-09-16T10:52:08.779 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:52:08.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: get_monmap_and_config 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: build_initial_monmap 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: monmap: 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:08.897501+0000 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:08.897501+0000 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:08.898 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:08.899 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 Processor -- start 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- start start 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: init 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39b009d6d0 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:08.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39b009e1c0 0x7f39b009f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-a con 0x7f39b009e1c0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39b0138c60 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00a0050 con 0x7f39b0138c60 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b009fd70 con 0x7f39b009d6d0 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00a0330 con 0x7f39b009e1c0 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient(hunting): authenticate will time out at 987.473904s 2024-09-16T10:52:08.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 0x7f39b013b050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57228/0 (socket says 172.21.15.5:57228) 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39affff700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f39b009d6d0 legacy=0x7f39b009dab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39b009e1c0 0x7f39b009f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 -- 172.21.15.5:0/446840797 learned_addr learned my addr 172.21.15.5:0/446840797 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): get_auth_request con 0x7f39b009e1c0 auth_method 0 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 -- 172.21.15.5:0/446840797 >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 legacy=0x7f39b013b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T10:52:08.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 -- 172.21.15.5:0/446840797 >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 legacy=0x7f39b013b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 --1- 172.21.15.5:0/446840797 >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 0x7f39b013b050 :-1 s=OPENED pgs=6606 cs=1 l=1).handle_message read tag failed 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b4bd9700 1 --1- 172.21.15.5:0/446840797 >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 0x7f39b013b050 :-1 s=OPENED pgs=6606 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.5:6789/0 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient(hunting): handle_auth_done global_id 28011 payload 274 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 -- 172.21.15.5:0/446840797 >> v1:172.21.15.143:6789/0 conn(0x7f39b009d6d0 legacy=0x7f39b009dab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 -- 172.21.15.5:0/446840797 >> v1:172.21.15.5:6789/0 conn(0x7f39b0138c60 legacy=0x7f39b013b050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 monclient: found mon.noname-a 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 -- 172.21.15.5:0/446840797 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00a18c0 con 0x7f39b009e1c0 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39af7fe700 1 --2- 172.21.15.5:0/446840797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b009f3f0 secure :-1 s=READY pgs=6607 cs=0 l=1 rev1=1 crypto rx=0x7f39b006b9a0 tx=0x7f39a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a123ff64801af308 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:08.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 1 -- 172.21.15.5:0/446840797 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a800e070 con 0x7f39b009e1c0 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: dump: 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: _finish_auth 0 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: _check_auth_tickets 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 5 monclient: authenticate success, global_id 28011 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 1 -- 172.21.15.5:0/446840797 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a800bb10 con 0x7f39b009e1c0 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 4 set_mon_vals no callback set 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 1 -- 172.21.15.5:0/446840797 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8003a40 con 0x7f39b009e1c0 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39aeffd700 10 monclient: dump: 2024-09-16T10:52:08.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 10 monclient: shutdown 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- 172.21.15.5:0/446840797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 msgr2=0x7f39b009f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 --2- 172.21.15.5:0/446840797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b009f3f0 secure :-1 s=READY pgs=6607 cs=0 l=1 rev1=1 crypto rx=0x7f39b006b9a0 tx=0x7f39a800b330 comp rx=0 tx=0).stop 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- 172.21.15.5:0/446840797 shutdown_connections 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 --2- 172.21.15.5:0/446840797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b009f3f0 unknown :-1 s=CLOSED pgs=6607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.896+0000 7f39b663c700 1 -- 172.21.15.5:0/446840797 >> 172.21.15.5:0/446840797 conn(0x7f39b00a0b30 msgr2=0x7f39b00a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 -- 172.21.15.5:0/446840797 shutdown_connections 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 -- 172.21.15.5:0/446840797 wait complete. 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient: build_initial_monmap 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient: monmap: 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:08.909 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:08.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 librados: starting msgr at 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 librados: starting objecter 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 Processor -- start 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 -- start start 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 librados: setting wanted keys 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 librados: calling monclient init 2024-09-16T10:52:08.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient: init 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 0x7f39b0133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.900+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-b con 0x7f39b009d6d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-a con 0x7f39b009e1c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 0x7f39b0133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): picked mon.noname-c con 0x7f39b0138c60 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b011f0b0 con 0x7f39b009e1c0 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:08.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0070280 con 0x7f39b009d6d0 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b009b1b0 con 0x7f39b0138c60 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient(hunting): authenticate will time out at 987.480213s 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39affff700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 msgr2=0x7f39b0133280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39affff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 0x7f39b0133280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:08.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:08.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 0x7f39b0133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40570/0 (socket says 172.21.15.5:40570) 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 -- 172.21.15.5:0/3407245966 learned_addr learned my addr 172.21.15.5:0/3407245966 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): get_auth_request con 0x7f39b009e1c0 auth_method 0 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): get_auth_request con 0x7f39b0138c60 auth_method 0 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:08.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient(hunting): handle_auth_done global_id 28014 payload 824 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 -- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 msgr2=0x7f39b0133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 0x7f39b0133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 -- 172.21.15.5:0/3407245966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 msgr2=0x7f39b0133280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 0x7f39b0133280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 monclient: found mon.noname-a 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:08.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 -- 172.21.15.5:0/3407245966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8009a70 con 0x7f39b009e1c0 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b4bd9700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 0x7f39b0133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 client.?.objecter ms_handle_connect 0x7f39b009e1c0 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39af7fe700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 secure :-1 s=READY pgs=6608 cs=0 l=1 rev1=1 crypto rx=0x7f39a800b730 tx=0x7f39a8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8012630 con 0x7f39b009e1c0 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: dump: 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.922 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: _finish_auth 0 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: _check_auth_tickets 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8012790 con 0x7f39b009e1c0 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 5 monclient: authenticate success, global_id 28014 2024-09-16T10:52:08.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _renew_subs 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8012b40 con 0x7f39b009e1c0 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b006c550 con 0x7f39b009e1c0 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 10 monclient: dump: 2024-09-16T10:52:08.924 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39ae7fc700 4 set_mon_vals no callback set 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 client.28014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _renew_subs 2024-09-16T10:52:08.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0139400 con 0x7f39b009e1c0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 librados: init done 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b009c640 con 0x7f39b009e1c0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f39a8012ca0 con 0x7f39b009e1c0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.904+0000 7f39acff9700 1 --2- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 0x7f39900393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7300+0+0 (secure 0 0 0) 0x7f39a804ab20 con 0x7f39b009e1c0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 10 client.28014.objecter ms_dispatch 0x7f39b0000fd0 osd_map(71..71 src has 1..71) v4 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 3 client.28014.objecter handle_osd_map got epochs [71,71] > 0 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 3 client.28014.objecter handle_osd_map decoding full epoch 71 2024-09-16T10:52:08.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 20 client.28014.objecter dump_active .. 0 homeless 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39affff700 1 --2- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 0x7f39900393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39affff700 10 monclient: get_auth_request con 0x7f3990036f10 auth_method 0 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39affff700 1 --2- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 0x7f39900393d0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f39980039e0 tx=0x7f3998009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.908+0000 7f39acff9700 10 client.28014.objecter ms_handle_connect 0x7f3990036f10 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.916+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f39a8011390 con 0x7f39b009e1c0 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.916+0000 7f39acff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:08.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:08.916+0000 7f39acff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:09.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39b663c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:52:09.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39b663c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:09.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39b663c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:52:09.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39b009f3f0 con 0x7f39b009e1c0 2024-09-16T10:52:09.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39acff9700 1 -- 172.21.15.5:0/3407245966 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7f39a801c3b0 con 0x7f39b009e1c0 2024-09-16T10:52:09.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39acff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:09.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.148+0000 7f39acff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:09.153 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 10 librados: watch_flush enter 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 10 librados: watch_flush exit 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 20 client.28014.objecter shutdown clearing up homeless session... 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 10 client.28014.objecter successfully canceled tick 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 msgr2=0x7f39900393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:09.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 0x7f39900393d0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f39980039e0 tx=0x7f3998009270 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 10 monclient: shutdown 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 msgr2=0x7f39b01337c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 secure :-1 s=READY pgs=6608 cs=0 l=1 rev1=1 crypto rx=0x7f39a800b730 tx=0x7f39a8004af0 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 shutdown_connections 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39b0138c60 0x7f39b0133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3990036f10 0x7f39900393d0 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39b009e1c0 0x7f39b01337c0 unknown :-1 s=CLOSED pgs=6608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 --2- 172.21.15.5:0/3407245966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b009d6d0 0x7f39b0133280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.152+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 >> 172.21.15.5:0/3407245966 conn(0x7f39b00a0b30 msgr2=0x7f39b00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.156+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 shutdown_connections 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.156+0000 7f39b663c700 1 -- 172.21.15.5:0/3407245966 wait complete. 2024-09-16T10:52:09.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:09.156+0000 7f39b663c700 1 librados: shutdown 2024-09-16T10:52:12.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:52:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: get_monmap_and_config 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: build_initial_monmap 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: monmap: 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:12.293725+0000 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:12.293725+0000 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:12.295 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 Processor -- start 2024-09-16T10:52:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 -- start start 2024-09-16T10:52:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: init 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf9409b4c0 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-a con 0x7faf9409e2c0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf94137c50 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9409c4f0 con 0x7faf9409b4c0 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf940a07f0 con 0x7faf94137c50 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9409dc00 con 0x7faf9409e2c0 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf9abfa700 10 monclient(hunting): authenticate will time out at 990.870113s 2024-09-16T10:52:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf98996700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7faf9409b4c0 0x7faf9409b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57234/0 (socket says 172.21.15.5:57234) 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf98996700 1 -- 172.21.15.5:0/3626193795 learned_addr learned my addr 172.21.15.5:0/3626193795 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.143:6789/0 conn(0x7faf94137c50 legacy=0x7faf9413a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf9409e2c0 auth_method 0 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:12.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.143:6789/0 conn(0x7faf94137c50 legacy=0x7faf9413a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf9409e2c0 auth_method 0 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:52:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.143:6789/0 conn(0x7faf94137c50 legacy=0x7faf9413a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf9409e2c0 auth_method 0 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:12.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:52:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.143:6789/0 conn(0x7faf94137c50 legacy=0x7faf9413a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf9409e2c0 auth_method 0 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:13.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): tick 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 msgr2=0x7faf9409bfb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf9409e2c0 0x7faf9409bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.143:6789/0 conn(0x7faf94137c50 legacy=0x7faf9413a040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.5:6789/0 conn(0x7faf9409b4c0 legacy=0x7faf9409b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient: _add_conns ranks=[4,2] 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): picked mon.noname-c con 0x7faf80005930 addr v2:172.21.15.143:3300/0 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): picked mon.noname-b con 0x7faf80008440 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9409dc00 con 0x7faf80005930 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8000b010 con 0x7faf80008440 2024-09-16T10:52:15.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf98996700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 msgr2=0x7faf80007db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf98996700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf80008440 auth_method 0 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:15.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf98996700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 msgr2=0x7faf80007db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf98996700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf80008440 auth_method 0 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:15.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.496+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:52:15.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf98996700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 msgr2=0x7faf80007db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:15.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf98996700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:52:15.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf80008440 auth_method 0 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:15.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:15.900+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:52:16.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf98996700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 msgr2=0x7faf80007db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:16.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf98996700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf80008440 auth_method 0 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:52:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:16.704+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): tick 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 msgr2=0x7faf8000a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80008440 0x7faf8000a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 msgr2=0x7faf80007db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:18.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.143:3300/0 conn(0x7faf80005930 0x7faf80007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 reap_dead start 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf9409b4c0 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf800088d0 0x7faf80005d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): picked mon.noname-a con 0x7faf800088d0 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf80006700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): picked mon.noname-b con 0x7faf80006250 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9409c960 con 0x7faf9409b4c0 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9409dc00 con 0x7faf800088d0 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 1 -- 172.21.15.5:0/3626193795 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8000b010 con 0x7faf80006250 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf80006700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): get_auth_request con 0x7faf80006250 auth_method 0 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf800088d0 0x7faf80005d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request con 0x7faf800088d0 auth_method 0 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.292+0000 7faf8bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 10 monclient(hunting): handle_auth_done global_id 22223 payload 274 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 1 -- 172.21.15.5:0/3626193795 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775350001 0 0) 0x7faf9409c960 con 0x7faf9409b4c0 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf800088d0 msgr2=0x7faf80005d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf800088d0 0x7faf80005d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 >> v1:172.21.15.5:6789/0 conn(0x7faf9409b4c0 legacy=0x7faf8000d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 monclient: found mon.noname-b 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 -- 172.21.15.5:0/3626193795 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf940a07f0 con 0x7faf80006250 2024-09-16T10:52:18.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf99197700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf80006700 secure :-1 s=READY pgs=4509 cs=0 l=1 rev1=1 crypto rx=0x7faf9406b9f0 tx=0x7faf8c00c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=684912520980f4eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 1 -- 172.21.15.5:0/3626193795 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c00e420 con 0x7faf80006250 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: dump: 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.303 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 1 -- 172.21.15.5:0/3626193795 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c00e580 con 0x7faf80006250 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 5 monclient: authenticate success, global_id 22223 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 1 -- 172.21.15.5:0/3626193795 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c00e930 con 0x7faf80006250 2024-09-16T10:52:18.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 4 set_mon_vals no callback set 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf8b7fe700 10 monclient: dump: 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.305 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 10 monclient: shutdown 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 -- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 msgr2=0x7faf80006700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf80006700 secure :-1 s=READY pgs=4509 cs=0 l=1 rev1=1 crypto rx=0x7faf9406b9f0 tx=0x7faf8c00c620 comp rx=0 tx=0).stop 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 -- 172.21.15.5:0/3626193795 shutdown_connections 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3626193795 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf80006700 unknown :-1 s=CLOSED pgs=4509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3626193795 >> v2:172.21.15.5:3300/0 conn(0x7faf800088d0 0x7faf80005d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 -- 172.21.15.5:0/3626193795 >> 172.21.15.5:0/3626193795 conn(0x7faf940a0b30 msgr2=0x7faf940a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 -- 172.21.15.5:0/3626193795 shutdown_connections 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.296+0000 7faf9abfa700 1 -- 172.21.15.5:0/3626193795 wait complete. 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient: build_initial_monmap 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient: monmap: 2024-09-16T10:52:18.306 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 librados: starting msgr at 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 librados: starting objecter 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 Processor -- start 2024-09-16T10:52:18.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 -- start start 2024-09-16T10:52:18.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 librados: setting wanted keys 2024-09-16T10:52:18.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 librados: calling monclient init 2024-09-16T10:52:18.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient: init 2024-09-16T10:52:18.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:18.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-c con 0x7faf80006250 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 0x7faf9412eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-b con 0x7faf800088d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 0x7faf9406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): picked mon.noname-a con 0x7faf9409b4c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf941c6eb0 con 0x7faf9409b4c0 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9409b1b0 con 0x7faf800088d0 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9406ef30 con 0x7faf80006250 2024-09-16T10:52:18.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf9abfa700 10 monclient(hunting): authenticate will time out at 996.878913s 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf8bfff700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 msgr2=0x7faf9412eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf8bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 0x7faf9412eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 0x7faf9406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33036/0 (socket says 172.21.15.5:33036) 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 1 -- 172.21.15.5:0/3748092123 learned_addr learned my addr 172.21.15.5:0/3748092123 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): get_auth_request con 0x7faf80006250 auth_method 0 2024-09-16T10:52:18.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): get_auth_request con 0x7faf9409b4c0 auth_method 0 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf99197700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.300+0000 7faf98996700 10 monclient(hunting): handle_auth_done global_id 22226 payload 824 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 -- 172.21.15.5:0/3748092123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 msgr2=0x7faf9412eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 0x7faf9412eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 -- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 msgr2=0x7faf9406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 0x7faf9406d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 monclient: found mon.noname-c 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 -- 172.21.15.5:0/3748092123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c007ae0 con 0x7faf80006250 2024-09-16T10:52:18.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf99197700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 0x7faf9406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf98996700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 secure :-1 s=READY pgs=4510 cs=0 l=1 rev1=1 crypto rx=0x7faf9000c730 tx=0x7faf9000ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 client.?.objecter ms_handle_connect 0x7faf80006250 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90004ca0 con 0x7faf80006250 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: dump: 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.316 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 5 monclient: authenticate success, global_id 22226 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90004e00 con 0x7faf80006250 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _renew_subs 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf940709f0 con 0x7faf80006250 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90007310 con 0x7faf80006250 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 monclient: dump: 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.318 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 client.22226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _renew_subs 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf8affd700 4 set_mon_vals no callback set 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf94070d80 con 0x7faf80006250 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 1 librados: init done 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7faf90007470 con 0x7faf80006250 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 --2- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 0x7faf8003cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7722+0+0 (secure 0 0 0) 0x7faf90048450 con 0x7faf80006250 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 10 client.22226.objecter ms_dispatch 0x7faf94000fd0 osd_map(73..73 src has 1..73) v4 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 3 client.22226.objecter handle_osd_map got epochs [73,73] > 0 2024-09-16T10:52:18.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 3 client.22226.objecter handle_osd_map decoding full epoch 73 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf897fa700 20 client.22226.objecter dump_active .. 0 homeless 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf94070b80 con 0x7faf80006250 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 0x7faf8003cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.304+0000 7faf8bfff700 10 monclient: get_auth_request con 0x7faf8003a840 auth_method 0 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.308+0000 7faf8bfff700 1 --2- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 0x7faf8003cd00 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7faf94070870 tx=0x7faf8401b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.308+0000 7faf897fa700 10 client.22226.objecter ms_handle_connect 0x7faf8003a840 2024-09-16T10:52:18.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.312+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faf90007720 con 0x7faf80006250 2024-09-16T10:52:18.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.312+0000 7faf897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.312+0000 7faf897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7faf9409b9c0 con 0x7faf80006250 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf897fa700 1 -- 172.21.15.5:0/3748092123 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7faf9001bb30 con 0x7faf80006250 2024-09-16T10:52:18.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:52:18.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:18.551 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":22,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:52:18.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 10 librados: watch_flush enter 2024-09-16T10:52:18.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.548+0000 7faf9abfa700 10 librados: watch_flush exit 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 20 client.22226.objecter shutdown clearing up homeless session... 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 10 client.22226.objecter successfully canceled tick 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 msgr2=0x7faf8003cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 0x7faf8003cd00 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7faf94070870 tx=0x7faf8401b040 comp rx=0 tx=0).stop 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 10 monclient: shutdown 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 msgr2=0x7faf941c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 secure :-1 s=READY pgs=4510 cs=0 l=1 rev1=1 crypto rx=0x7faf9000c730 tx=0x7faf9000ca40 comp rx=0 tx=0).stop 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 shutdown_connections 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> v2:172.21.15.143:6812/1098205034 conn(0x7faf8003a840 0x7faf8003cd00 unknown :-1 s=CLOSED pgs=3672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faf9409b4c0 0x7faf9406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf800088d0 0x7faf9412eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 --2- 172.21.15.5:0/3748092123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7faf80006250 0x7faf941c5f40 unknown :-1 s=CLOSED pgs=4510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 >> 172.21.15.5:0/3748092123 conn(0x7faf940a0b30 msgr2=0x7faf9412fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 shutdown_connections 2024-09-16T10:52:18.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 -- 172.21.15.5:0/3748092123 wait complete. 2024-09-16T10:52:18.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.552+0000 7faf9abfa700 1 librados: shutdown 2024-09-16T10:52:18.566 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T10:52:18.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T10:52:18.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:52:18.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.688+0000 7efcf8b50700 10 monclient: get_monmap_and_config 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.688+0000 7efcf8b50700 10 monclient: build_initial_monmap 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.688+0000 7efcf8b50700 10 monclient: monmap: 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:18.691558+0000 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:18.691558+0000 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:18.696 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:18.697 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:18.697 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:18.697 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 Processor -- start 2024-09-16T10:52:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 -- start start 2024-09-16T10:52:18.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient: init 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-b con 0x7efcf40a1650 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efcf409ee10 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcf409f210 addr v1:172.21.15.5:6789/0 2024-09-16T10:52:18.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf4132890 con 0x7efcf409f210 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf409b6d0 con 0x7efcf409ee10 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf409f660 con 0x7efcf40a1650 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf8b50700 10 monclient(hunting): authenticate will time out at 997.279008s 2024-09-16T10:52:18.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf1d9b700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7efcf409ee10 legacy=0x7efcf4131a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.700+0000 7efcf259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33040/0 (socket says 172.21.15.5:33040) 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 -- 172.21.15.5:0/974464246 learned_addr learned my addr 172.21.15.5:0/974464246 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): get_auth_request con 0x7efcf40a1650 auth_method 0 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient(hunting): handle_auth_done global_id 22229 payload 274 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 -- 172.21.15.5:0/974464246 >> v1:172.21.15.143:6789/0 conn(0x7efcf409ee10 legacy=0x7efcf4131a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 -- 172.21.15.5:0/974464246 >> v1:172.21.15.5:6789/0 conn(0x7efcf409f210 legacy=0x7efcf4132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 monclient: found mon.noname-b 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 -- 172.21.15.5:0/974464246 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf409b9b0 con 0x7efcf40a1650 2024-09-16T10:52:18.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.704+0000 7efcf259c700 1 --2- 172.21.15.5:0/974464246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 secure :-1 s=READY pgs=4511 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e80 tx=0x7efcec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abf9400a41d45e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 1 -- 172.21.15.5:0/974464246 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec00e070 con 0x7efcf40a1650 2024-09-16T10:52:18.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:52:18.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: dump: 2024-09-16T10:52:18.710 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.711 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.711 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.711 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.711 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.711 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 5 monclient: authenticate success, global_id 22229 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 1 -- 172.21.15.5:0/974464246 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec003a40 con 0x7efcf40a1650 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:18.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 4 set_mon_vals no callback set 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 1 -- 172.21.15.5:0/974464246 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec003dd0 con 0x7efcf40a1650 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf159a700 10 monclient: dump: 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.713 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: shutdown 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- 172.21.15.5:0/974464246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 msgr2=0x7efcf409aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 --2- 172.21.15.5:0/974464246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 secure :-1 s=READY pgs=4511 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e80 tx=0x7efcec00b330 comp rx=0 tx=0).stop 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- 172.21.15.5:0/974464246 shutdown_connections 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 --2- 172.21.15.5:0/974464246 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf40a1650 0x7efcf409aa50 unknown :-1 s=CLOSED pgs=4511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- 172.21.15.5:0/974464246 >> 172.21.15.5:0/974464246 conn(0x7efcf40a0b30 msgr2=0x7efcf40a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- 172.21.15.5:0/974464246 shutdown_connections 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- 172.21.15.5:0/974464246 wait complete. 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: build_initial_monmap 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: monmap: 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:18.714 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:18.715 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 librados: starting msgr at 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 librados: starting objecter 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 Processor -- start 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- start start 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 librados: setting wanted keys 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 librados: calling monclient init 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: init 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcf409ee10 0x7efcf41ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-b con 0x7efcf409ee10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-c con 0x7efcf409f210 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf40a1650 0x7efcf41cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): picked mon.noname-a con 0x7efcf40a1650 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf413bcd0 con 0x7efcf40a1650 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf409b1b0 con 0x7efcf409ee10 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf413a850 con 0x7efcf409f210 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf8b50700 10 monclient(hunting): authenticate will time out at 997.287502s 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf259c700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcf409ee10 msgr2=0x7efcf41ca6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:18.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcf409ee10 0x7efcf41ca6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33056/0 (socket says 172.21.15.5:33056) 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 1 -- 172.21.15.5:0/4268973322 learned_addr learned my addr 172.21.15.5:0/4268973322 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): get_auth_request con 0x7efcf409f210 auth_method 0 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf40a1650 0x7efcf41cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): get_auth_request con 0x7efcf40a1650 auth_method 0 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.708+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 10 monclient(hunting): handle_auth_done global_id 22232 payload 824 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcf409ee10 msgr2=0x7efcf41ca6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcf409ee10 0x7efcf41ca6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf40a1650 msgr2=0x7efcf41cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf40a1650 0x7efcf41cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 monclient: found mon.noname-c 2024-09-16T10:52:18.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec009b30 con 0x7efcf409f210 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf40a1650 0x7efcf41cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf1d9b700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 secure :-1 s=READY pgs=4512 cs=0 l=1 rev1=1 crypto rx=0x7efce800b780 tx=0x7efce800ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.?.objecter ms_handle_connect 0x7efcf409f210 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8003e70 con 0x7efcf409f210 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: dump: 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.725 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 5 monclient: authenticate success, global_id 22232 2024-09-16T10:52:18.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 monclient: _renew_subs 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce8005070 con 0x7efcf409f210 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf40a34a0 con 0x7efcf409f210 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8005420 con 0x7efcf409f210 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: dump: 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.727 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf0d99700 4 set_mon_vals no callback set 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 client.22232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 monclient: _renew_subs 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf41390a0 con 0x7efcf409f210 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 1 librados: init done 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7efce8005580 con 0x7efcf409f210 2024-09-16T10:52:18.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 --2- 172.21.15.5:0/4268973322 >> v2:172.21.15.143:6812/1098205034 conn(0x7efcd4036f10 0x7efcd40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6523+0+0 (secure 0 0 0) 0x7efce8047420 con 0x7efcf409f210 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.22232.objecter ms_dispatch 0x7efcf4000fd0 osd_map(74..74 src has 1..74) v4 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 3 client.22232.objecter handle_osd_map got epochs [74,74] > 0 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 3 client.22232.objecter handle_osd_map decoding full epoch 74 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 20 client.22232.objecter dump_active .. 0 homeless 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf259c700 1 --2- 172.21.15.5:0/4268973322 >> v2:172.21.15.143:6812/1098205034 conn(0x7efcd4036f10 0x7efcd40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf259c700 10 monclient: get_auth_request con 0x7efcd4036f10 auth_method 0 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 0x7efcf41317f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4067580 con 0x7efcf41341a0 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 0x7efcf41317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient: get_auth_request con 0x7efcf41341a0 auth_method 0 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.22232.objecter ms_handle_connect 0x7efcd4036f10 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf259c700 1 --2- 172.21.15.5:0/4268973322 >> v2:172.21.15.143:6812/1098205034 conn(0x7efcd4036f10 0x7efcd40393d0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e80 tx=0x7efcec009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_done global_id 28047 payload 824 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efcf409aa50 con 0x7efcf41341a0 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.22232.objecter ms_handle_connect 0x7efcf41341a0 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 0x7efcf41317f0 secure :-1 s=READY pgs=6621 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce400ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 client.22232.objecter resend_mon_ops 2024-09-16T10:52:18.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40052e0 con 0x7efcf41341a0 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: dump: 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.731 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7efce4005440 con 0x7efcf41341a0 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 msgr2=0x7efcf41317f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 0x7efcf41317f0 secure :-1 s=READY pgs=6621 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce400ff20 comp rx=0 tx=0).stop 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 msgr2=0x7efcf41317f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:18.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.712+0000 7efce2ffd700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf41341a0 0x7efcf41317f0 secure :-1 s=CLOSED pgs=6621 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce400ff20 comp rx=0 tx=0).stop 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf8b50700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf8b50700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 0x7efcf4132140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf8b50700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4132680 con 0x7efcf4131d30 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 0x7efcf4132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient: get_auth_request con 0x7efcf4131d30 auth_method 0 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 10 monclient(hunting): handle_auth_done global_id 28068 payload 824 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 1 -- 172.21.15.5:0/4268973322 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efcf41384e0 con 0x7efcf4131d30 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efce2ffd700 10 client.22232.objecter ms_handle_connect 0x7efcf4131d30 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efce2ffd700 10 client.22232.objecter resend_mon_ops 2024-09-16T10:52:18.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.724+0000 7efcf2d9d700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 0x7efcf4132140 secure :-1 s=READY pgs=6625 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce40011e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.756+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4022070 con 0x7efcf4131d30 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.756+0000 7efce2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.756+0000 7efce2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.756+0000 7efce2ffd700 10 monclient: dump: 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.758 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.759 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7efce401d280 con 0x7efcf4131d30 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 msgr2=0x7efcf4132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 0x7efcf4132140 secure :-1 s=READY pgs=6625 cs=0 l=1 rev1=1 crypto rx=0x7efce4007fd0 tx=0x7efce40011e0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 msgr2=0x7efcf4132140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:18.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efce2ffd700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcf4131d30 0x7efcf4132140 unknown :-1 s=CLOSED pgs=6625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 22, 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:52:18.797 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:52:18.798 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:52:18.799 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:52:18.800 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.801 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.802 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.803 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.804 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:52:18.805 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:52:18.806 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.807 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:52:18.808 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.809 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.810 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:52:18.810 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:52:18.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efcf8b50700 10 librados: watch_flush enter 2024-09-16T10:52:18.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efcf8b50700 10 librados: watch_flush exit 2024-09-16T10:52:18.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efcf8b50700 20 client.22232.objecter shutdown clearing up homeless session... 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.792+0000 7efcf8b50700 10 client.22232.objecter successfully canceled tick 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 >> v2:172.21.15.143:6812/1098205034 conn(0x7efcd4036f10 msgr2=0x7efcd40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 --2- 172.21.15.5:0/4268973322 >> v2:172.21.15.143:6812/1098205034 conn(0x7efcd4036f10 0x7efcd40393d0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e80 tx=0x7efcec009000 comp rx=0 tx=0).stop 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 10 monclient: shutdown 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf2d9d700 1 -- 172.21.15.5:0/4268973322 reap_dead start 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 msgr2=0x7efcf41cac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 secure :-1 s=READY pgs=4512 cs=0 l=1 rev1=1 crypto rx=0x7efce800b780 tx=0x7efce800ba90 comp rx=0 tx=0).stop 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 shutdown_connections 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 --2- 172.21.15.5:0/4268973322 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efcf409f210 0x7efcf41cac00 unknown :-1 s=CLOSED pgs=4512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 >> 172.21.15.5:0/4268973322 conn(0x7efcf40a0b30 msgr2=0x7efcf409d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 shutdown_connections 2024-09-16T10:52:18.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 -- 172.21.15.5:0/4268973322 wait complete. 2024-09-16T10:52:18.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.796+0000 7efcf8b50700 1 librados: shutdown 2024-09-16T10:52:18.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:52:18.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T10:52:18.813 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:52:18.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.924+0000 7f10d7fff700 10 monclient: get_monmap_and_config 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.924+0000 7f10d7fff700 10 monclient: build_initial_monmap 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.924+0000 7f10d7fff700 10 monclient: monmap: 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:52:18.928798+0000 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:52:18.928798+0000 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:52:18.929 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 Processor -- start 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 -- start start 2024-09-16T10:52:18.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient: init 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f10d80a0970 0x7f10d80a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f10d80a0970 addr v2:172.21.15.5:3300/0 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10d809ca50 addr v1:172.21.15.143:6789/0 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 0x7f10d8132e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f10d80a1290 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d809a440 con 0x7f10d809ca50 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d809f4f0 con 0x7f10d80a0970 2024-09-16T10:52:18.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8099d80 con 0x7f10d80a1290 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f10d80a0970 0x7f10d80a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d67fc700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f10d809ca50 legacy=0x7f10d8132720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f10d80a0970 0x7f10d80a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:56944/0 (socket says 172.21.15.5:56944) 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 -- 172.21.15.5:0/394583136 learned_addr learned my addr 172.21.15.5:0/394583136 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 0x7f10d8132e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d7fff700 10 monclient(hunting): authenticate will time out at 997.506889s 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): get_auth_request con 0x7f10d80a0970 auth_method 0 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): get_auth_request con 0x7f10d80a1290 auth_method 0 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_done global_id 28071 payload 274 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 -- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 msgr2=0x7f10d8132e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:52:18.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 0x7f10d8132e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 -- 172.21.15.5:0/394583136 >> v1:172.21.15.143:6789/0 conn(0x7f10d809ca50 legacy=0x7f10d8132720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 monclient: found mon.noname-a 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d6ffd700 1 -- 172.21.15.5:0/394583136 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d809a1c0 con 0x7f10d80a0970 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.928+0000 7f10d77fe700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 0x7f10d8132e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a0970 0x7f10d80a0d50 secure :-1 s=READY pgs=6626 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009910 tx=0x7f10c8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626006761311ed43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 1 -- 172.21.15.5:0/394583136 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c8004230 con 0x7f10d80a0970 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:52:18.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: dump: 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 5 monclient: authenticate success, global_id 28071 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 1 -- 172.21.15.5:0/394583136 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c8004390 con 0x7f10d80a0970 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 4 set_mon_vals no callback set 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 1 -- 172.21.15.5:0/394583136 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80036c0 con 0x7f10d80a0970 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d5ffb700 10 monclient: dump: 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.939 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: shutdown 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a0970 msgr2=0x7f10d80a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a0970 0x7f10d80a0d50 secure :-1 s=READY pgs=6626 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009910 tx=0x7f10c8009c20 comp rx=0 tx=0).stop 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- 172.21.15.5:0/394583136 shutdown_connections 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d80a1290 0x7f10d8132e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- 172.21.15.5:0/394583136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a0970 0x7f10d80a0d50 unknown :-1 s=CLOSED pgs=6626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- 172.21.15.5:0/394583136 >> 172.21.15.5:0/394583136 conn(0x7f10d80a2090 msgr2=0x7f10d806a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- 172.21.15.5:0/394583136 shutdown_connections 2024-09-16T10:52:18.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- 172.21.15.5:0/394583136 wait complete. 2024-09-16T10:52:18.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: build_initial_monmap 2024-09-16T10:52:18.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: monmap: 2024-09-16T10:52:18.946 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:52:18.946 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:52:18.946 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 librados: starting msgr at 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 librados: starting objecter 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 Processor -- start 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- start start 2024-09-16T10:52:18.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 librados: setting wanted keys 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 librados: calling monclient init 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: init 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f10d809ca50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d80a0970 0x7f10d81cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f10d80a0970 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a1290 0x7f10d813b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f10d80a1290 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8138c80 con 0x7f10d80a1290 2024-09-16T10:52:18.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d809c150 con 0x7f10d80a0970 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80a3fd0 con 0x7f10d809ca50 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d7fff700 10 monclient(hunting): authenticate will time out at 997.510922s 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a1290 0x7f10d813b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d67fc700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d80a0970 msgr2=0x7f10d81cb1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33072/0 (socket says 172.21.15.5:33072) 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 -- 172.21.15.5:0/212162735 learned_addr learned my addr 172.21.15.5:0/212162735 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d67fc700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d80a0970 0x7f10d81cb1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): get_auth_request con 0x7f10d809ca50 auth_method 0 2024-09-16T10:52:18.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): get_auth_request con 0x7f10d80a1290 auth_method 0 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient(hunting): handle_auth_done global_id 22250 payload 824 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d80a0970 msgr2=0x7f10d81cb1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10d80a0970 0x7f10d81cb1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a1290 msgr2=0x7f10d813b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10d80a1290 0x7f10d813b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 monclient: found mon.noname-c 2024-09-16T10:52:18.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0009a70 con 0x7f10d809ca50 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10d6ffd700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 secure :-1 s=READY pgs=4516 cs=0 l=1 rev1=1 crypto rx=0x7f10c8004000 tx=0x7f10c8004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 10 client.?.objecter ms_handle_connect 0x7f10d809ca50 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c8003ea0 con 0x7f10d809ca50 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.932+0000 7f10bffff700 10 monclient: dump: 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.953 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: _finish_auth 0 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: _check_auth_tickets 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 5 monclient: authenticate success, global_id 22250 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c8011630 con 0x7f10d809ca50 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 monclient: _renew_subs 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c8009570 con 0x7f10d809ca50 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80119e0 con 0x7f10d809ca50 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: dump: 2024-09-16T10:52:18.955 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d57fa700 4 set_mon_vals no callback set 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 client.22250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:52:18.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 monclient: _renew_subs 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d813afe0 con 0x7f10d809ca50 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 1 librados: init done 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 0x7f10d813bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d804e890 con 0x7f10d813b940 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 0x7f10d813bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient: get_auth_request con 0x7f10d813b940 auth_method 0 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f10c8021020 con 0x7f10d809ca50 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 10 monclient(hunting): handle_auth_done global_id 22253 payload 824 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10d80a16a0 con 0x7f10d813b940 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d67fc700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 0x7f10d813bd20 secure :-1 s=READY pgs=4517 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0039e0 tx=0x7f10cc009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 --2- 172.21.15.5:0/212162735 >> v2:172.21.15.143:6812/1098205034 conn(0x7f10c4036f10 0x7f10c40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 client.22250.objecter ms_handle_connect 0x7f10d813b940 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 client.22250.objecter resend_mon_ops 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6523+0+0 (secure 0 0 0) 0x7f10c8049540 con 0x7f10d809ca50 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 client.22250.objecter ms_dispatch 0x7f10d8000fd0 osd_map(74..74 src has 1..74) v4 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 3 client.22250.objecter handle_osd_map got epochs [74,74] > 0 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 3 client.22250.objecter handle_osd_map decoding full epoch 74 2024-09-16T10:52:18.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 20 client.22250.objecter dump_active .. 0 homeless 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc003ab0 con 0x7f10d813b940 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: dump: 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.959 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f10cc003c10 con 0x7f10d813b940 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d77fe700 1 --2- 172.21.15.5:0/212162735 >> v2:172.21.15.143:6812/1098205034 conn(0x7f10c4036f10 0x7f10c40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 msgr2=0x7f10d813bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 0x7f10d813bd20 secure :-1 s=READY pgs=4517 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0039e0 tx=0x7f10cc009f30 comp rx=0 tx=0).stop 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 msgr2=0x7f10d813bd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813b940 0x7f10d813bd20 unknown :-1 s=CLOSED pgs=4517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d77fe700 10 monclient: get_auth_request con 0x7f10c4036f10 auth_method 0 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10bffff700 10 client.22250.objecter ms_handle_connect 0x7f10c4036f10 2024-09-16T10:52:18.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.936+0000 7f10d77fe700 1 --2- 172.21.15.5:0/212162735 >> v2:172.21.15.143:6812/1098205034 conn(0x7f10c4036f10 0x7f10c40393d0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f10d0011580 tx=0x7f10d000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 0x7f10d806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d809b470 con 0x7f10d813c260 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 0x7f10d806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient: get_auth_request con 0x7f10d813c260 auth_method 0 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 10 monclient(hunting): handle_auth_done global_id 22256 payload 824 2024-09-16T10:52:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 1 -- 172.21.15.5:0/212162735 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10d806ac90 con 0x7f10d813c260 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d67fc700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 0x7f10d806f060 secure :-1 s=READY pgs=4518 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc005ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 client.22250.objecter ms_handle_connect 0x7f10d813c260 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 client.22250.objecter resend_mon_ops 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc01b800 con 0x7f10d813c260 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 monclient: dump: 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:52:18.964 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7f10cc015470 con 0x7f10d813c260 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 msgr2=0x7f10d806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 0x7f10d806f060 secure :-1 s=READY pgs=4518 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc005ad0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 msgr2=0x7f10d806f060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:52:18.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10bffff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d813c260 0x7f10d806f060 secure :-1 s=CLOSED pgs=4518 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009fd0 tx=0x7f10cc005ad0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 22, 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:18.966 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:52:18.967 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.968 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:52:18.969 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:52:18.970 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.971 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.972 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.973 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:52:18.974 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:52:18.975 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.976 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 8 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.977 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:52:18.978 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 librados: watch_flush enter 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 librados: watch_flush exit 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 20 client.22250.objecter shutdown clearing up homeless session... 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 client.22250.objecter successfully canceled tick 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 >> v2:172.21.15.143:6812/1098205034 conn(0x7f10c4036f10 msgr2=0x7f10c40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 --2- 172.21.15.5:0/212162735 >> v2:172.21.15.143:6812/1098205034 conn(0x7f10c4036f10 0x7f10c40393d0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f10d0011580 tx=0x7f10d000b040 comp rx=0 tx=0).stop 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 10 monclient: shutdown 2024-09-16T10:52:18.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 msgr2=0x7f10d81cac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d77fe700 1 -- 172.21.15.5:0/212162735 reap_dead start 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 secure :-1 s=READY pgs=4516 cs=0 l=1 rev1=1 crypto rx=0x7f10c8004000 tx=0x7f10c8004770 comp rx=0 tx=0).stop 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 shutdown_connections 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 --2- 172.21.15.5:0/212162735 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10d809ca50 0x7f10d81cac80 unknown :-1 s=CLOSED pgs=4516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 >> 172.21.15.5:0/212162735 conn(0x7f10d80a2090 msgr2=0x7f10d809e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 shutdown_connections 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 -- 172.21.15.5:0/212162735 wait complete. 2024-09-16T10:52:18.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:52:18.948+0000 7f10d7fff700 1 librados: shutdown 2024-09-16T10:52:18.981 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:52:18.982 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:52:20.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-16T10:52:20.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T10:52:20.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T10:52:20.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2003 ms) 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2047 ms) 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2001 ms) 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2018 ms) 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8069 ms total) 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T10:52:20.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4040 ms) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3986 ms) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5202 ms) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13228 ms total) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2993 ms) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2003 ms) 2024-09-16T10:52:20.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2014 ms) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1987 ms) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8997 ms total) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4042 ms) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (18063 ms) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (22105 ms total) 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: 2024-09-16T10:52:20.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T10:52:20.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (72477 ms total) 2024-09-16T10:52:20.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T10:52:22.696 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T10:52:22.696 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T10:52:22.696 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T10:52:22.696 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T10:52:22.696 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: seed 24491 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (157 ms) 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (9 ms) 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T10:52:22.697 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (16 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (17 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-09-16T10:52:22.698 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (9 ms) 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (21 ms) 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3680 ms) 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (295 ms) 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T10:52:22.699 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: done waiting 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60494 ms) 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (11 ms) 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (7189 ms) 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T10:52:22.700 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T10:52:22.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-16T10:52:22.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T10:52:22.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T10:52:31.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: seed 24607 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1990 ms) 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2039 ms) 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2011 ms) 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T10:52:31.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2016 ms) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3852 ms) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11908 ms total) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4017 ms) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4000 ms) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5984 ms) 2024-09-16T10:52:31.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4029 ms) 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2011 ms) 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: /build/ceph-18.2.4-845-ge4e227b3/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi143-24607-7 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17985 ms) 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38027 ms total) 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:31.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (6 ms) 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4930 ms) 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4936 ms total) 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3012 ms) 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2004 ms) 2024-09-16T10:52:31.710 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T10:52:46.744 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (20 api_aio: Running main() from gmock_main.cc 2024-09-16T10:52:46.744 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T10:52:46.744 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T10:52:46.744 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2669 ms) 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (2995 ms) 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3029 ms) 2024-09-16T10:52:46.745 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2842 ms) 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3014 ms) 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2991 ms) 2024-09-16T10:52:46.746 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2997 ms) 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3014 ms) 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2988 ms) 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T10:52:46.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3007 ms) 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3003 ms) 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3000 ms) 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T10:52:46.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3009 ms) 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3000 ms) 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.Flush (3005 ms) 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T10:52:46.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2992 ms) 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3021 ms) 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3010 ms) 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (16975 ms) 2024-09-16T10:52:46.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3005 ms) 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3034 ms) 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2978 ms) 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T10:52:46.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3005 ms) 2024-09-16T10:52:46.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T10:52:46.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2974 ms) 2024-09-16T10:52:46.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T10:52:46.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3021 ms) 2024-09-16T10:52:46.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T10:52:46.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2994 ms) 2024-09-16T10:52:46.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 26 tests from LibRadosAio (91576 ms total) 2024-09-16T10:52:46.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: 2024-09-16T10:52:46.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T10:52:46.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout:P (71935 ms total) 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (110 ms) 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (110 ms total) 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T10:52:48.701 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (27 ms) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (2 ms) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (29 ms total) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (23 ms) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (26 ms total) 2024-09-16T10:52:48.702 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (20 ms) 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (23 ms total) 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1029 ms) 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1029 ms total) 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: 2024-09-16T10:52:48.703 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T10:52:48.704 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (100490 ms total) 2024-09-16T10:52:48.704 INFO:tasks.workunit.client.0.smithi143.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout:08 ms) 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1975 ms) 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8999 ms total) 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T10:52:54.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4027 ms) 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3983 ms) 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4011 ms) 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12021 ms total) 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (106473 ms total) 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T10:52:54.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T10:53:02.915 INFO:tasks.workunit.client.0.smithi143.stdout:pi_list: : cursor()=6:62a1935d:::14:head expected=6:62a1935d:::14:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:62a1935d:::14:head -> 14 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=14 expected=14 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:5c6b0b28:::7:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:5c6b0b28:::7:head expected=6:5c6b0b28:::7:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:5c6b0b28:::7:head -> 7 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=7 expected=7 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:566253c9:::13:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:566253c9:::13:head expected=6:566253c9:::13:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=13 expected=13 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-09-16T10:53:02.916 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:f905c69b:::2:head expected=6:f905c69b:::2:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:f905c69b:::2:head -> 2 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=2 expected=2 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:e960b815:::9:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:e960b815:::9:head expected=6:e960b815:::9:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:e960b815:::9:head -> 9 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=9 expected=9 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:bd63b0f1:::8:head expected=6:bd63b0f1:::8:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=8 expected=8 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-09-16T10:53:02.917 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:b29083e3:::5:head expected=6:b29083e3:::5:head 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=5 expected=5 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : cursor()=6:89d3ae78:::11:head expected=6:89d3ae78:::11:head 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: : entry=11 expected=11 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (465 ms) 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (83081 ms) 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 0/5 -> MIN 6:33333333::::head 2024-09-16T10:53:02.918 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 1/5 -> 6:33333333::::head 6:66666666::::head 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 2/5 -> 6:66666666::::head 6:99999999::::head 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 3/5 -> 6:99999999::::head 6:cccccccc::::head 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: split 4/5 -> 6:cccccccc::::head MAX 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (23982 ms) 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 7 tests from LibRadosList (108041 ms total) 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1072 ms) 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-09-16T10:53:02.919 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo2 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo3 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo4 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo5 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo1 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset foo6,foo7 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo7 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: foo6 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo4 2024-09-16T10:53:02.920 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo5 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo7 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns2:foo6 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: ns1:foo1 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo1 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo2 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: :foo3 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (70 ms) 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 1 0 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 10 0 2024-09-16T10:53:02.921 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 13 0 2024-09-16T10:53:03.704 INFO:tasks.workunit.client.0.smithi143.stdout: api_lis api_aio_pp: Running main() from gmock_main.cc 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2659 ms) 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19867 ms) 2024-09-16T10:53:03.705 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6023 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3033 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2965 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3001 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2992 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3006 ms) 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T10:53:03.706 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3035 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2985 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3009 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (17967 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3015 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3036 ms) 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T10:53:03.707 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2972 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3004 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2989 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3006 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2997 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3018 ms) 2024-09-16T10:53:03.708 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2994 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3024 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2990 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2980 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3012 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3042 ms) 2024-09-16T10:53:03.709 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T10:53:08.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAit: 7 0 2024-09-16T10:53:08.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 14 0 2024-09-16T10:53:08.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 0 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 15 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 11 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 5 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 8 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 6 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 3 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 4 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 12 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 9 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2 0 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (100 ms) 2024-09-16T10:53:08.753 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 3 tests from LibRadosListEC (1242 ms total) 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3037 ms) 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] 1 test from LibRadosListNP (3037 ms total) 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [==========] 11 tests from 3 test suites ran. (120580 ms total) 2024-09-16T10:53:08.754 INFO:tasks.workunit.client.0.smithi143.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T10:53:48.982 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:53:48.983 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T10:53:48.983 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:53:48.984 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:53:48.987 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:53:48.987 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:53:48.988 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: get_monmap_and_config 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: build_initial_monmap 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: monmap: 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:49.110 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:49.109824+0000 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:49.109824+0000 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:49.111 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 Processor -- start 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 -- start start 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: init 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a809e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f59a809aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:49.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59a8071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:49.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-c con 0x7f59a81376e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:49.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a809caa0 con 0x7f59a8071870 2024-09-16T10:53:49.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a809e3a0 con 0x7f59a81376e0 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a809ae30 con 0x7f59a809aa50 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a759e700 10 monclient(hunting): authenticate will time out at 1087.687131s 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a809e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a5d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f59a8071870 0x7f59a809ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:32872/0 (socket says 172.21.15.5:32872) 2024-09-16T10:53:49.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a5d9b700 1 -- 172.21.15.5:0/1254314740 learned_addr learned my addr 172.21.15.5:0/1254314740 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 10 monclient(hunting): get_auth_request con 0x7f59a81376e0 auth_method 0 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): get_auth_request con 0x7f59a809aa50 auth_method 0 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 1 -- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 msgr2=0x7f59a8139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 1 -- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 msgr2=0x7f59a8139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:53:49.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 1 --2- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a6d9d700 1 --2- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.106+0000 7f59a659c700 10 monclient(hunting): handle_auth_done global_id 22529 payload 274 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 -- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 msgr2=0x7f59a8139ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 --2- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 -- 172.21.15.5:0/1254314740 >> v1:172.21.15.143:6789/0 conn(0x7f59a8071870 legacy=0x7f59a809ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient: _finish_hunting 0 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 monclient: found mon.noname-b 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 -- 172.21.15.5:0/1254314740 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a809b5f0 con 0x7f59a809aa50 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 --2- 172.21.15.5:0/1254314740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a809e8e0 secure :-1 s=READY pgs=4615 cs=0 l=1 rev1=1 crypto rx=0x7f5998009850 tx=0x7f5998009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ffd8db6ded1fb24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 1 -- 172.21.15.5:0/1254314740 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011830 con 0x7f59a809aa50 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: dump: 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:49.127 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: _finish_auth 0 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: _check_auth_tickets 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 5 monclient: authenticate success, global_id 22529 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:49.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 1 -- 172.21.15.5:0/1254314740 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998011990 con 0x7f59a809aa50 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 4 set_mon_vals no callback set 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 1 -- 172.21.15.5:0/1254314740 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998011d20 con 0x7f59a809aa50 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:49.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a559a700 10 monclient: dump: 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: shutdown 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:49.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- 172.21.15.5:0/1254314740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 msgr2=0x7f59a809e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- 172.21.15.5:0/1254314740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a809e8e0 secure :-1 s=READY pgs=4615 cs=0 l=1 rev1=1 crypto rx=0x7f5998009850 tx=0x7f5998009b60 comp rx=0 tx=0).stop 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- 172.21.15.5:0/1254314740 shutdown_connections 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- 172.21.15.5:0/1254314740 >> v2:172.21.15.143:3300/0 conn(0x7f59a81376e0 0x7f59a8139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- 172.21.15.5:0/1254314740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a809e8e0 unknown :-1 s=CLOSED pgs=4615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- 172.21.15.5:0/1254314740 >> 172.21.15.5:0/1254314740 conn(0x7f59a80a45d0 msgr2=0x7f59a80a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- 172.21.15.5:0/1254314740 shutdown_connections 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- 172.21.15.5:0/1254314740 wait complete. 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: build_initial_monmap 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: monmap: 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:49.131 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 librados: starting msgr at 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 librados: starting objecter 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 Processor -- start 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- start start 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 librados: setting wanted keys 2024-09-16T10:53:49.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 librados: calling monclient init 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: init 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 0x7f59a81ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-a con 0x7f59a8071870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-c con 0x7f59a809aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f59a81376e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80a31c0 con 0x7f59a8071870 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80a2d00 con 0x7f59a81376e0 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a809b0d0 con 0x7f59a809aa50 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a759e700 10 monclient(hunting): authenticate will time out at 1087.691482s 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 0x7f59a81ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 0x7f59a81ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57338/0 (socket says 172.21.15.5:57338) 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:36502/0 (socket says 172.21.15.5:36502) 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 1 -- 172.21.15.5:0/3641329358 learned_addr learned my addr 172.21.15.5:0/3641329358 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): get_auth_request con 0x7f59a8071870 auth_method 0 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:49.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:49.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:49.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): get_auth_request con 0x7f59a809aa50 auth_method 0 2024-09-16T10:53:49.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:49.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:49.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a6d9d700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a6d9d700 10 monclient(hunting): get_auth_request con 0x7f59a81376e0 auth_method 0 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:49.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.110+0000 7f59a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a6d9d700 1 -- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 msgr2=0x7f59a8137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a6d9d700 1 -- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 msgr2=0x7f59a8137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a6d9d700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a6d9d700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 10 monclient(hunting): handle_auth_done global_id 22532 payload 824 2024-09-16T10:53:49.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 -- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 msgr2=0x7f59a8137060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 -- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 msgr2=0x7f59a81ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 0x7f59a81ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 monclient: found mon.noname-c 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 -- 172.21.15.5:0/3641329358 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980094b0 con 0x7f59a809aa50 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a5d9b700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 secure :-1 s=READY pgs=4616 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c450 tx=0x7f599c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 client.?.objecter ms_handle_connect 0x7f59a809aa50 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:49.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c005d90 con 0x7f59a809aa50 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: dump: 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:49.140 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: _finish_auth 0 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 5 monclient: authenticate success, global_id 22532 2024-09-16T10:53:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c0070d0 con 0x7f59a809aa50 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _renew_subs 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a81201c0 con 0x7f59a809aa50 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c007440 con 0x7f59a809aa50 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 monclient: dump: 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 client.22532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _renew_subs 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a813dad0 con 0x7f59a809aa50 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a4d99700 4 set_mon_vals no callback set 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 1 librados: init done 2024-09-16T10:53:49.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f599c0075a0 con 0x7f59a809aa50 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a811fc00 con 0x7f59a809aa50 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 --2- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 0x7f5988039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6515+0+0 (secure 0 0 0) 0x7f599c047390 con 0x7f59a809aa50 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 client.22532.objecter ms_dispatch 0x7f59a8000fd0 osd_map(164..164 src has 1..164) v4 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 3 client.22532.objecter handle_osd_map got epochs [164,164] > 0 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 3 client.22532.objecter handle_osd_map decoding full epoch 164 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 20 client.22532.objecter dump_active .. 0 homeless 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a659c700 1 --2- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 0x7f5988039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:49.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a659c700 10 monclient: get_auth_request con 0x7f5988036f90 auth_method 0 2024-09-16T10:53:49.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f5996ffd700 10 client.22532.objecter ms_handle_connect 0x7f5988036f90 2024-09-16T10:53:49.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.114+0000 7f59a659c700 1 --2- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 0x7f5988039450 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f5998009ba0 tx=0x7f5998008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:49.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.122+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f599c01b3b0 con 0x7f59a809aa50 2024-09-16T10:53:49.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.122+0000 7f5996ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:49.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.122+0000 7f5996ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:49.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f59a759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:53:49.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f59a759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:49.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f59a759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:49.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59a809caa0 con 0x7f59a809aa50 2024-09-16T10:53:49.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f5996ffd700 1 -- 172.21.15.5:0/3641329358 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f59a809caa0 con 0x7f59a809aa50 2024-09-16T10:53:49.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f5996ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:49.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.358+0000 7f5996ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:49.363 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":22,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:53:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 10 librados: watch_flush enter 2024-09-16T10:53:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 10 librados: watch_flush exit 2024-09-16T10:53:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 20 client.22532.objecter shutdown clearing up homeless session... 2024-09-16T10:53:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 10 client.22532.objecter successfully canceled tick 2024-09-16T10:53:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 msgr2=0x7f5988039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 0x7f5988039450 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f5998009ba0 tx=0x7f5998008030 comp rx=0 tx=0).stop 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 10 monclient: shutdown 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 msgr2=0x7f59a8136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 secure :-1 s=READY pgs=4616 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c450 tx=0x7f599c00c760 comp rx=0 tx=0).stop 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 shutdown_connections 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59a81376e0 0x7f59a8137060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59a809aa50 0x7f59a8136b20 unknown :-1 s=CLOSED pgs=4616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5988036f90 0x7f5988039450 unknown :-1 s=CLOSED pgs=3732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 --2- 172.21.15.5:0/3641329358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59a8071870 0x7f59a81ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 >> 172.21.15.5:0/3641329358 conn(0x7f59a80a45d0 msgr2=0x7f59a813a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 shutdown_connections 2024-09-16T10:53:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 -- 172.21.15.5:0/3641329358 wait complete. 2024-09-16T10:53:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:49.362+0000 7f59a759e700 1 librados: shutdown 2024-09-16T10:53:52.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: get_monmap_and_config 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: build_initial_monmap 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: monmap: 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.501 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:52.501254+0000 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:52.501254+0000 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:52.502 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 Processor -- start 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 -- start start 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: init 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 0x7f41140a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): picked mon.noname-b con 0x7f41140a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 0x7f4114135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): picked mon.noname-a con 0x7f41140a35a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4114135f80 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140a3db0 con 0x7f4114135f80 2024-09-16T10:53:52.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41140a4b70 con 0x7f41140a35a0 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140a0850 con 0x7f41140a2c80 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient(hunting): authenticate will time out at 1091.077609s 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 0x7f4114135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 0x7f41140a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 0x7f4114135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60742/0 (socket says 172.21.15.5:60742) 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 1 -- 172.21.15.5:0/3482253228 learned_addr learned my addr 172.21.15.5:0/3482253228 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): get_auth_request con 0x7f41140a35a0 auth_method 0 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): get_auth_request con 0x7f41140a2c80 auth_method 0 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient(hunting): handle_auth_done global_id 22541 payload 274 2024-09-16T10:53:52.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 -- 172.21.15.5:0/3482253228 >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 msgr2=0x7f4114135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 --2- 172.21.15.5:0/3482253228 >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 0x7f4114135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 -- 172.21.15.5:0/3482253228 >> v1:172.21.15.5:6789/0 conn(0x7f4114135f80 legacy=0x7f411413cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 monclient: found mon.noname-b 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 -- 172.21.15.5:0/3482253228 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140a4180 con 0x7f41140a2c80 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4113fff700 1 --2- 172.21.15.5:0/3482253228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 0x7f41140a3060 secure :-1 s=READY pgs=4619 cs=0 l=1 rev1=1 crypto rx=0x7f4104009850 tx=0x7f4104009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9988b51556303e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 1 -- 172.21.15.5:0/3482253228 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104011830 con 0x7f41140a2c80 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:53:52.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: dump: 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: _finish_auth 0 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:53:52.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 1 -- 172.21.15.5:0/3482253228 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104011990 con 0x7f41140a2c80 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 5 monclient: authenticate success, global_id 22541 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 1 -- 172.21.15.5:0/3482253228 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104011d40 con 0x7f41140a2c80 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4119435700 4 set_mon_vals no callback set 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.498+0000 7f4112ffd700 10 monclient: dump: 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.511 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: shutdown 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- 172.21.15.5:0/3482253228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 msgr2=0x7f41140a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- 172.21.15.5:0/3482253228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 0x7f41140a3060 secure :-1 s=READY pgs=4619 cs=0 l=1 rev1=1 crypto rx=0x7f4104009850 tx=0x7f4104009b60 comp rx=0 tx=0).stop 2024-09-16T10:53:52.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- 172.21.15.5:0/3482253228 shutdown_connections 2024-09-16T10:53:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- 172.21.15.5:0/3482253228 >> v2:172.21.15.5:3300/0 conn(0x7f41140a35a0 0x7f4114135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- 172.21.15.5:0/3482253228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a2c80 0x7f41140a3060 unknown :-1 s=CLOSED pgs=4619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- 172.21.15.5:0/3482253228 >> 172.21.15.5:0/3482253228 conn(0x7f41140a4d80 msgr2=0x7f41140a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- 172.21.15.5:0/3482253228 shutdown_connections 2024-09-16T10:53:52.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- 172.21.15.5:0/3482253228 wait complete. 2024-09-16T10:53:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: build_initial_monmap 2024-09-16T10:53:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: monmap: 2024-09-16T10:53:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:52.518 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 librados: starting msgr at 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 librados: starting objecter 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 Processor -- start 2024-09-16T10:53:52.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- start start 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 librados: setting wanted keys 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 librados: calling monclient init 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: init 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): picked mon.noname-a con 0x7f41140a2c80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 0x7f4114121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): picked mon.noname-c con 0x7f41140a35a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4114135f80 0x7f4114122330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): picked mon.noname-b con 0x7f4114135f80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4114123f70 con 0x7f41140a2c80 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41141254b0 con 0x7f4114135f80 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411409f4e0 con 0x7f41140a35a0 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.502+0000 7f4119435700 10 monclient(hunting): authenticate will time out at 1091.083892s 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 0x7f4114121df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60746/0 (socket says 172.21.15.5:60746) 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 0x7f4114121df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39074/0 (socket says 172.21.15.5:39074) 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 -- 172.21.15.5:0/516192013 learned_addr learned my addr 172.21.15.5:0/516192013 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:52.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): get_auth_request con 0x7f41140a2c80 auth_method 0 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): get_auth_request con 0x7f41140a35a0 auth_method 0 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4118c34700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4114135f80 0x7f4114122330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4118c34700 10 monclient(hunting): get_auth_request con 0x7f4114135f80 auth_method 0 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4118c34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4118c34700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4118c34700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient(hunting): handle_auth_done global_id 28431 payload 824 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 -- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 msgr2=0x7f4114121df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 0x7f4114121df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 -- 172.21.15.5:0/516192013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4114135f80 msgr2=0x7f4114122330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4114135f80 0x7f4114122330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 monclient: found mon.noname-a 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 -- 172.21.15.5:0/516192013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040094b0 con 0x7f41140a2c80 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4113fff700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 secure :-1 s=READY pgs=6736 cs=0 l=1 rev1=1 crypto rx=0x7f41040090b0 tx=0x7f4104004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 client.?.objecter ms_handle_connect 0x7f41140a2c80 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040036c0 con 0x7f41140a2c80 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: dump: 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.524 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: _finish_auth 0 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 5 monclient: authenticate success, global_id 28431 2024-09-16T10:53:52.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104003820 con 0x7f41140a2c80 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _renew_subs 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4114125db0 con 0x7f41140a2c80 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104003bd0 con 0x7f41140a2c80 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 client.28431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f41127fc700 4 set_mon_vals no callback set 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 monclient: dump: 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _renew_subs 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140a72b0 con 0x7f41140a2c80 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 1 librados: init done 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140700b0 con 0x7f41140a2c80 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4104021070 con 0x7f41140a2c80 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 --2- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 0x7f4100039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5788+0+0 (secure 0 0 0) 0x7f4104049580 con 0x7f41140a2c80 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 10 client.28431.objecter ms_dispatch 0x7f4114000fd0 osd_map(167..167 src has 1..167) v4 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 3 client.28431.objecter handle_osd_map got epochs [167,167] > 0 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.506+0000 7f4110ff9700 3 client.28431.objecter handle_osd_map decoding full epoch 167 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.510+0000 7f4110ff9700 20 client.28431.objecter dump_active .. 0 homeless 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.510+0000 7f41137fe700 1 --2- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 0x7f4100039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.510+0000 7f41137fe700 10 monclient: get_auth_request con 0x7f4100036f60 auth_method 0 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.510+0000 7f4110ff9700 10 client.28431.objecter ms_handle_connect 0x7f4100036f60 2024-09-16T10:53:52.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.510+0000 7f41137fe700 1 --2- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 0x7f4100039420 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f41141353c0 tx=0x7f41080092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.514+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4104010380 con 0x7f41140a2c80 2024-09-16T10:53:52.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.514+0000 7f4110ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:52.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.514+0000 7f4110ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:52.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4119435700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:53:52.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4119435700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:52.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4119435700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41141247e0 con 0x7f41140a2c80 2024-09-16T10:53:52.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4110ff9700 1 -- 172.21.15.5:0/516192013 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f41141247e0 con 0x7f41140a2c80 2024-09-16T10:53:52.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4110ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:52.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.754+0000 7f4110ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:52.758 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:53:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 10 librados: watch_flush enter 2024-09-16T10:53:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 10 librados: watch_flush exit 2024-09-16T10:53:52.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 20 client.28431.objecter shutdown clearing up homeless session... 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 10 client.28431.objecter successfully canceled tick 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 msgr2=0x7f4100039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 0x7f4100039420 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f41141353c0 tx=0x7f41080092e0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 10 monclient: shutdown 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 msgr2=0x7f4114135620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 secure :-1 s=READY pgs=6736 cs=0 l=1 rev1=1 crypto rx=0x7f41040090b0 tx=0x7f4104004770 comp rx=0 tx=0).stop 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 shutdown_connections 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4114135f80 0x7f4114122330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f41140a35a0 0x7f4114121df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4100036f60 0x7f4100039420 unknown :-1 s=CLOSED pgs=3734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 --2- 172.21.15.5:0/516192013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f41140a2c80 0x7f4114135620 unknown :-1 s=CLOSED pgs=6736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 >> 172.21.15.5:0/516192013 conn(0x7f41140a4d80 msgr2=0x7f41141bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 shutdown_connections 2024-09-16T10:53:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 -- 172.21.15.5:0/516192013 wait complete. 2024-09-16T10:53:52.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.758+0000 7f4119435700 1 librados: shutdown 2024-09-16T10:53:52.777 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:53:52.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:53:52.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:53:52.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: get_monmap_and_config 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: build_initial_monmap 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: monmap: 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:52.890115+0000 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:52.890115+0000 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 Processor -- start 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 -- start start 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: init 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87340a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-b con 0x7f87340a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87340a4280 0x7f87340a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-a con 0x7f87340a4280 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87340a3d10 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873409fdc0 con 0x7f87340a3d10 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87340a1e50 con 0x7f87340a4280 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873409fae0 con 0x7f87340a2eb0 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 10 monclient(hunting): authenticate will time out at 1091.466445s 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87340a4280 0x7f87340a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87340a4280 0x7f87340a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60760/0 (socket says 172.21.15.5:60760) 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 -- 172.21.15.5:0/3166033805 learned_addr learned my addr 172.21.15.5:0/3166033805 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): get_auth_request con 0x7f87340a4280 auth_method 0 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87340a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): get_auth_request con 0x7f87340a2eb0 auth_method 0 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient(hunting): handle_auth_done global_id 28440 payload 274 2024-09-16T10:53:52.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 -- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 msgr2=0x7f87340a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87340a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 -- 172.21.15.5:0/3166033805 >> v1:172.21.15.143:6789/0 conn(0x7f87340a3d10 legacy=0x7f8734135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 monclient: found mon.noname-a 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 -- 172.21.15.5:0/3166033805 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873409efb0 con 0x7f87340a4280 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f87340a37d0 secure :-1 s=READY pgs=6737 cs=0 l=1 rev1=1 crypto rx=0x7f8724009910 tx=0x7f8724009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0f70eed943aebe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 1 -- 172.21.15.5:0/3166033805 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724011830 con 0x7f87340a4280 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 10 monclient: dump: 2024-09-16T10:53:52.898 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 10 monclient: _finish_auth 0 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 10 monclient: _check_auth_tickets 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f87327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:52.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 5 monclient: authenticate success, global_id 28440 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.886+0000 7f8739abb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 1 -- 172.21.15.5:0/3166033805 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8724011990 con 0x7f87340a4280 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 1 -- 172.21.15.5:0/3166033805 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724011d40 con 0x7f87340a4280 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 4 set_mon_vals no callback set 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f87327fc700 10 monclient: dump: 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 10 monclient: shutdown 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 -- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 msgr2=0x7f87340a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f87340a37d0 secure :-1 s=READY pgs=6737 cs=0 l=1 rev1=1 crypto rx=0x7f8724009910 tx=0x7f8724009c20 comp rx=0 tx=0).stop 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 -- 172.21.15.5:0/3166033805 shutdown_connections 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f87340a37d0 unknown :-1 s=CLOSED pgs=6737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 --2- 172.21.15.5:0/3166033805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87340a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 -- 172.21.15.5:0/3166033805 >> 172.21.15.5:0/3166033805 conn(0x7f87340a4fb0 msgr2=0x7f87340a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 -- 172.21.15.5:0/3166033805 shutdown_connections 2024-09-16T10:53:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 1 -- 172.21.15.5:0/3166033805 wait complete. 2024-09-16T10:53:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 10 monclient: build_initial_monmap 2024-09-16T10:53:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.890+0000 7f8739abb700 10 monclient: monmap: 2024-09-16T10:53:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 librados: starting msgr at 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 librados: starting objecter 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 Processor -- start 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 -- start start 2024-09-16T10:53:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 librados: setting wanted keys 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 librados: calling monclient init 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient: init 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-c con 0x7f87340a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87340a3d10 0x7f87341caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-b con 0x7f87340a3d10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f8734122570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): picked mon.noname-a con 0x7f87340a4280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8734126570 con 0x7f87340a4280 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8734124de0 con 0x7f87340a3d10 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87340a77b0 con 0x7f87340a2eb0 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8739abb700 10 monclient(hunting): authenticate will time out at 1091.474997s 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39090/0 (socket says 172.21.15.5:39090) 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 -- 172.21.15.5:0/3809120749 learned_addr learned my addr 172.21.15.5:0/3809120749 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): get_auth_request con 0x7f87340a2eb0 auth_method 0 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87340a3d10 0x7f87341caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f8734122570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): get_auth_request con 0x7f87340a3d10 auth_method 0 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): get_auth_request con 0x7f87340a4280 auth_method 0 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8733fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient(hunting): handle_auth_done global_id 22559 payload 824 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87340a3d10 msgr2=0x7f87341caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87340a3d10 0x7f87341caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 msgr2=0x7f8734122570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87340a4280 0x7f8734122570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient: _finish_hunting 0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 monclient: found mon.noname-c 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.894+0000 7f87337fe700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724009570 con 0x7f87340a2eb0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f87337fe700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 secure :-1 s=READY pgs=4622 cs=0 l=1 rev1=1 crypto rx=0x7f872800ebe0 tx=0x7f872800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.?.objecter ms_handle_connect 0x7f87340a2eb0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728009740 con 0x7f87340a2eb0 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: dump: 2024-09-16T10:53:52.914 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: _finish_auth 0 2024-09-16T10:53:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872800c850 con 0x7f87340a2eb0 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 5 monclient: authenticate success, global_id 22559 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 monclient: _renew_subs 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87340a7240 con 0x7f87340a2eb0 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872800cbe0 con 0x7f87340a2eb0 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: dump: 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 client.22559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 monclient: _renew_subs 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8731ffb700 4 set_mon_vals no callback set 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8734126290 con 0x7f87340a2eb0 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 1 librados: init done 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f872800cd40 con 0x7f87340a2eb0 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 0x7f8734122e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8734067b80 con 0x7f8734122ab0 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 --2- 172.21.15.5:0/3809120749 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8720036f60 0x7f8720039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4916+0+0 (secure 0 0 0) 0x7f8728047cb0 con 0x7f87340a2eb0 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.22559.objecter ms_dispatch 0x7f8734000fd0 osd_map(168..168 src has 1..168) v4 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 3 client.22559.objecter handle_osd_map got epochs [168,168] > 0 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 3 client.22559.objecter handle_osd_map decoding full epoch 168 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 0x7f8734122e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient: get_auth_request con 0x7f8734122ab0 auth_method 0 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 20 client.22559.objecter dump_active .. 0 homeless 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8733fff700 1 --2- 172.21.15.5:0/3809120749 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8720036f60 0x7f8720039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8733fff700 10 monclient: get_auth_request con 0x7f8720036f60 auth_method 0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 10 monclient(hunting): handle_auth_done global_id 28449 payload 824 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8734136380 con 0x7f8734122ab0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.22559.objecter ms_handle_connect 0x7f8734122ab0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 0x7f8734122e90 secure :-1 s=READY pgs=6739 cs=0 l=1 rev1=1 crypto rx=0x7f8724009910 tx=0x7f8724004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.22559.objecter resend_mon_ops 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724020070 con 0x7f8734122ab0 2024-09-16T10:53:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: dump: 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 client.22559.objecter ms_handle_connect 0x7f8720036f60 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f8733fff700 1 --2- 172.21.15.5:0/3809120749 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8720036f60 0x7f8720039420 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 crypto rx=0x7f872c0012e0 tx=0x7f872c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f87240040c0 con 0x7f8734122ab0 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 msgr2=0x7f8734122e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 0x7f8734122e90 secure :-1 s=READY pgs=6739 cs=0 l=1 rev1=1 crypto rx=0x7f8724009910 tx=0x7f8724004770 comp rx=0 tx=0).stop 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 msgr2=0x7f8734122e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.898+0000 7f871bfff700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8734122ab0 0x7f8734122e90 unknown :-1 s=CLOSED pgs=6739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 0x7f873411eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87341367b0 con 0x7f87341233d0 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 0x7f873411eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient: get_auth_request con 0x7f87341233d0 auth_method 0 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 10 monclient(hunting): handle_auth_done global_id 28455 payload 824 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 1 -- 172.21.15.5:0/3809120749 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8734124210 con 0x7f87341233d0 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 client.22559.objecter ms_handle_connect 0x7f87341233d0 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8732ffd700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 0x7f873411eb00 secure :-1 s=READY pgs=6741 cs=0 l=1 rev1=1 crypto rx=0x7f8724000c00 tx=0x7f8724011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 client.22559.objecter resend_mon_ops 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724003ce0 con 0x7f87341233d0 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 monclient: dump: 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:52.926 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f8724003e40 con 0x7f87341233d0 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:52.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 msgr2=0x7f873411eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 0x7f873411eb00 secure :-1 s=READY pgs=6741 cs=0 l=1 rev1=1 crypto rx=0x7f8724000c00 tx=0x7f8724011000 comp rx=0 tx=0).stop 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 msgr2=0x7f873411eb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f871bfff700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87341233d0 0x7f873411eb00 unknown :-1 s=CLOSED pgs=6741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:52.928 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:52.929 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:52.930 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:52.931 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:52.932 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:52.933 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.934 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.935 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.936 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:52.937 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:52.938 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.939 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:52.940 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:53:52.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:52.942 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:52.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:52.943 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:52.943 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:52.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 librados: watch_flush enter 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 librados: watch_flush exit 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 20 client.22559.objecter shutdown clearing up homeless session... 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 client.22559.objecter successfully canceled tick 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8720036f60 msgr2=0x7f8720039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 --2- 172.21.15.5:0/3809120749 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8720036f60 0x7f8720039420 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 crypto rx=0x7f872c0012e0 tx=0x7f872c009040 comp rx=0 tx=0).stop 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 10 monclient: shutdown 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8733fff700 1 -- 172.21.15.5:0/3809120749 reap_dead start 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 msgr2=0x7f87341caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:52.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 secure :-1 s=READY pgs=4622 cs=0 l=1 rev1=1 crypto rx=0x7f872800ebe0 tx=0x7f872800eef0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 shutdown_connections 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 --2- 172.21.15.5:0/3809120749 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f87340a2eb0 0x7f87341caa30 unknown :-1 s=CLOSED pgs=4622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 >> 172.21.15.5:0/3809120749 conn(0x7f87340a4fb0 msgr2=0x7f87340a79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 shutdown_connections 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 -- 172.21.15.5:0/3809120749 wait complete. 2024-09-16T10:53:52.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:52.910+0000 7f8739abb700 1 librados: shutdown 2024-09-16T10:53:52.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:52.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:53:52.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.062+0000 7f23830da700 10 monclient: get_monmap_and_config 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.062+0000 7f23830da700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.062+0000 7f23830da700 10 monclient: monmap: 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:53.068763+0000 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:53.068763+0000 2024-09-16T10:53:53.069 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 Processor -- start 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- start start 2024-09-16T10:53:53.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: init 2024-09-16T10:53:53.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T10:53:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f237c09aa50 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f237c071870 0x7f237c09eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): picked mon.noname-a con 0x7f237c071870 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:53.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 0x7f237c139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): picked mon.noname-c con 0x7f237c1376e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c09caa0 con 0x7f237c09aa50 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c09e3e0 con 0x7f237c071870 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c09ae30 con 0x7f237c1376e0 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient(hunting): authenticate will time out at 1091.645094s 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f237c071870 0x7f237c09eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f237c071870 0x7f237c09eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60806/0 (socket says 172.21.15.5:60806) 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 -- 172.21.15.5:0/3628846392 learned_addr learned my addr 172.21.15.5:0/3628846392 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): get_auth_request con 0x7f237c071870 auth_method 0 2024-09-16T10:53:53.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23828d9700 1 --2- 172.21.15.5:0/3628846392 >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 0x7f237c139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient(hunting): handle_auth_done global_id 28458 payload 274 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 -- 172.21.15.5:0/3628846392 >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 msgr2=0x7f237c139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 --2- 172.21.15.5:0/3628846392 >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 0x7f237c139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 -- 172.21.15.5:0/3628846392 >> v1:172.21.15.5:6789/0 conn(0x7f237c09aa50 legacy=0x7f237c09e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 monclient: found mon.noname-a 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 -- 172.21.15.5:0/3628846392 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c09b5f0 con 0x7f237c071870 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23828d9700 10 monclient: get_auth_request con 0x7f237c1376e0 auth_method 0 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23828d9700 1 --2- 172.21.15.5:0/3628846392 >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 0x7f237c139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:53:53.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23818d7700 1 --2- 172.21.15.5:0/3628846392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c071870 0x7f237c09eff0 secure :-1 s=READY pgs=6743 cs=0 l=1 rev1=1 crypto rx=0x7f237c09cf00 tx=0x7f2378009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99e76ef7eae5f71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 1 -- 172.21.15.5:0/3628846392 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237800b8a0 con 0x7f237c071870 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: dump: 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.098 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.099 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 5 monclient: authenticate success, global_id 28458 2024-09-16T10:53:53.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 1 -- 172.21.15.5:0/3628846392 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237800ba00 con 0x7f237c071870 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 4 set_mon_vals no callback set 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 1 -- 172.21.15.5:0/3628846392 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237800bd90 con 0x7f237c071870 2024-09-16T10:53:53.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:53.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23810d6700 10 monclient: dump: 2024-09-16T10:53:53.102 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.102 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.103 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.104 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: shutdown 2024-09-16T10:53:53.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- 172.21.15.5:0/3628846392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c071870 msgr2=0x7f237c09eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 --2- 172.21.15.5:0/3628846392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c071870 0x7f237c09eff0 secure :-1 s=READY pgs=6743 cs=0 l=1 rev1=1 crypto rx=0x7f237c09cf00 tx=0x7f2378009bf0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- 172.21.15.5:0/3628846392 shutdown_connections 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 --2- 172.21.15.5:0/3628846392 >> v2:172.21.15.143:3300/0 conn(0x7f237c1376e0 0x7f237c139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 --2- 172.21.15.5:0/3628846392 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c071870 0x7f237c09eff0 unknown :-1 s=CLOSED pgs=6743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- 172.21.15.5:0/3628846392 >> 172.21.15.5:0/3628846392 conn(0x7f237c0a45d0 msgr2=0x7f237c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- 172.21.15.5:0/3628846392 shutdown_connections 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 -- 172.21.15.5:0/3628846392 wait complete. 2024-09-16T10:53:53.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 10 monclient: monmap: 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:53.106 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.066+0000 7f23830da700 1 librados: starting msgr at 2024-09-16T10:53:53.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 librados: starting objecter 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 Processor -- start 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- start start 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 librados: setting wanted keys 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 librados: calling monclient init 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: init 2024-09-16T10:53:53.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c071870 0x7f237c11ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): picked mon.noname-b con 0x7f237c071870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 0x7f237c11f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): picked mon.noname-c con 0x7f237c09aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:53.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c1376e0 0x7f237c11f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): picked mon.noname-a con 0x7f237c1376e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c070530 con 0x7f237c1376e0 2024-09-16T10:53:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1367f0 con 0x7f237c071870 2024-09-16T10:53:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c09b0d0 con 0x7f237c09aa50 2024-09-16T10:53:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient(hunting): authenticate will time out at 1091.649095s 2024-09-16T10:53:53.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 0x7f237c11f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 0x7f237c11f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39092/0 (socket says 172.21.15.5:39092) 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 -- 172.21.15.5:0/1529509158 learned_addr learned my addr 172.21.15.5:0/1529509158 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): get_auth_request con 0x7f237c09aa50 auth_method 0 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c1376e0 0x7f237c11f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): get_auth_request con 0x7f237c1376e0 auth_method 0 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c071870 0x7f237c11ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 10 monclient(hunting): get_auth_request con 0x7f237c071870 auth_method 0 2024-09-16T10:53:53.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23828d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient(hunting): handle_auth_done global_id 22565 payload 824 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c071870 msgr2=0x7f237c11ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c071870 0x7f237c11ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c1376e0 msgr2=0x7f237c11f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f237c1376e0 0x7f237c11f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 monclient: found mon.noname-c 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378009570 con 0x7f237c09aa50 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c071870 0x7f237c11ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23818d7700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 0x7f237c11f270 secure :-1 s=READY pgs=4623 cs=0 l=1 rev1=1 crypto rx=0x7f2378000c00 tx=0x7f2378004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 client.?.objecter ms_handle_connect 0x7f237c09aa50 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23780038c0 con 0x7f237c09aa50 2024-09-16T10:53:53.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: dump: 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.119 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 5 monclient: authenticate success, global_id 22565 2024-09-16T10:53:53.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378003a20 con 0x7f237c09aa50 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _renew_subs 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c0a34d0 con 0x7f237c09aa50 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2378003db0 con 0x7f237c09aa50 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:53.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f236effd700 10 monclient: dump: 2024-09-16T10:53:53.123 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.123 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.123 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 client.22565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _renew_subs 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c122150 con 0x7f237c09aa50 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 1 librados: init done 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23808d5700 4 set_mon_vals no callback set 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.070+0000 7f23830da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23830da700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 0x7f237c061580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c067380 con 0x7f237c02c820 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 0x7f237c061580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient: get_auth_request con 0x7f237c02c820 auth_method 0 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2378020070 con 0x7f237c09aa50 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 10 monclient(hunting): handle_auth_done global_id 22568 payload 824 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f237c09d580 con 0x7f237c02c820 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 --2- 172.21.15.5:0/1529509158 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2360036f90 0x7f2360039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4916+0+0 (secure 0 0 0) 0x7f23780490a0 con 0x7f237c09aa50 2024-09-16T10:53:53.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 client.22565.objecter ms_dispatch 0x7f237c000fd0 osd_map(168..168 src has 1..168) v4 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 3 client.22565.objecter handle_osd_map got epochs [168,168] > 0 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 3 client.22565.objecter handle_osd_map decoding full epoch 168 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 20 client.22565.objecter dump_active .. 0 homeless 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 0x7f237c061580 secure :-1 s=READY pgs=4624 cs=0 l=1 rev1=1 crypto rx=0x7f237000e4f0 tx=0x7f237000e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 client.22565.objecter ms_handle_connect 0x7f237c02c820 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 client.22565.objecter resend_mon_ops 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23828d9700 1 --2- 172.21.15.5:0/1529509158 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2360036f90 0x7f2360039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23828d9700 10 monclient: get_auth_request con 0x7f2360036f90 auth_method 0 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700052e0 con 0x7f237c02c820 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:53.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 monclient: dump: 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2370005440 con 0x7f237c02c820 2024-09-16T10:53:53.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 msgr2=0x7f237c061580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 0x7f237c061580 secure :-1 s=READY pgs=4624 cs=0 l=1 rev1=1 crypto rx=0x7f237000e4f0 tx=0x7f237000e8b0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 msgr2=0x7f237c061580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c02c820 0x7f237c061580 secure :-1 s=CLOSED pgs=4624 cs=0 l=1 rev1=1 crypto rx=0x7f237000e4f0 tx=0x7f237000e8b0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f236effd700 10 client.22565.objecter ms_handle_connect 0x7f2360036f90 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.074+0000 7f23828d9700 1 --2- 172.21.15.5:0/1529509158 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2360036f90 0x7f2360039450 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7f237c137040 tx=0x7f2374007780 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 0x7f237c1d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c09c250 con 0x7f237c061ac0 2024-09-16T10:53:53.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 0x7f237c1d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient: get_auth_request con 0x7f237c061ac0 auth_method 0 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 10 monclient(hunting): handle_auth_done global_id 22571 payload 824 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 1 -- 172.21.15.5:0/1529509158 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f237c0a0680 con 0x7f237c061ac0 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 client.22565.objecter ms_handle_connect 0x7f237c061ac0 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 client.22565.objecter resend_mon_ops 2024-09-16T10:53:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23820d8700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 0x7f237c1d08c0 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 crypto rx=0x7f2370015000 tx=0x7f2370000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2370022070 con 0x7f237c061ac0 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 monclient: dump: 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.131 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:53:53.132 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.133 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:53.134 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.135 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:53.136 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.138 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.139 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.140 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.141 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.142 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.144 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:53:53.145 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:53.146 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f23700036a0 con 0x7f237c061ac0 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 msgr2=0x7f237c1d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 0x7f237c1d08c0 secure :-1 s=READY pgs=4625 cs=0 l=1 rev1=1 crypto rx=0x7f2370015000 tx=0x7f2370000ed0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 msgr2=0x7f237c1d08c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f236effd700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c061ac0 0x7f237c1d08c0 unknown :-1 s=CLOSED pgs=4625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 librados: watch_flush enter 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 librados: watch_flush exit 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 20 client.22565.objecter shutdown clearing up homeless session... 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 client.22565.objecter successfully canceled tick 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2360036f90 msgr2=0x7f2360039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 --2- 172.21.15.5:0/1529509158 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2360036f90 0x7f2360039450 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7f237c137040 tx=0x7f2374007780 comp rx=0 tx=0).stop 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 10 monclient: shutdown 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 msgr2=0x7f237c11f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 --2- 172.21.15.5:0/1529509158 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f237c09aa50 0x7f237c11f270 secure :-1 s=READY pgs=4623 cs=0 l=1 rev1=1 crypto rx=0x7f2378000c00 tx=0x7f2378004770 comp rx=0 tx=0).stop 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23828d9700 1 -- 172.21.15.5:0/1529509158 reap_dead start 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23828d9700 1 -- 172.21.15.5:0/1529509158 reap_dead start 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 shutdown_connections 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 >> 172.21.15.5:0/1529509158 conn(0x7f237c0a45d0 msgr2=0x7f237c06fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 shutdown_connections 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 -- 172.21.15.5:0/1529509158 wait complete. 2024-09-16T10:53:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.086+0000 7f23830da700 1 librados: shutdown 2024-09-16T10:53:53.151 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:53.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:53:53.151 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:53:53.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.254+0000 7f8813fff700 10 monclient: get_monmap_and_config 2024-09-16T10:53:53.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.254+0000 7f8813fff700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.254+0000 7f8813fff700 10 monclient: monmap: 2024-09-16T10:53:53.260 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.260 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:53.260155+0000 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:53.260155+0000 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:53.261 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 Processor -- start 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- start start 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient: init 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 0x7f88140a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-b con 0x7f88140a1ad0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88140a1eb0 0x7f88140a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-c con 0x7f88140a1eb0 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:53.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8814139c90 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881409de20 con 0x7f8814139c90 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881409fc70 con 0x7f88140a1eb0 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881409e020 con 0x7f88140a1ad0 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient(hunting): authenticate will time out at 1091.836856s 2024-09-16T10:53:53.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 0x7f88140a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 0x7f88140a0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39114/0 (socket says 172.21.15.5:39114) 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 1 -- 172.21.15.5:0/2353201140 learned_addr learned my addr 172.21.15.5:0/2353201140 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): get_auth_request con 0x7f88140a1ad0 auth_method 0 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 --2- 172.21.15.5:0/2353201140 >> v2:172.21.15.143:3300/0 conn(0x7f88140a1eb0 0x7f88140a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): get_auth_request con 0x7f88140a1eb0 auth_method 0 2024-09-16T10:53:53.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient(hunting): handle_auth_done global_id 44107 payload 274 2024-09-16T10:53:53.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 -- 172.21.15.5:0/2353201140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 msgr2=0x7f88140a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 --2- 172.21.15.5:0/2353201140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 0x7f88140a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 -- 172.21.15.5:0/2353201140 >> v1:172.21.15.143:6789/0 conn(0x7f8814139c90 legacy=0x7f881413c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 monclient: found mon.noname-c 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 -- 172.21.15.5:0/2353201140 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881409f340 con 0x7f88140a1eb0 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f88127fc700 1 --2- 172.21.15.5:0/2353201140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88140a1eb0 0x7f88140a22b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8808009850 tx=0x7f8808009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a4562ea19af0113 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 1 -- 172.21.15.5:0/2353201140 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808011830 con 0x7f88140a1eb0 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: dump: 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.271 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 5 monclient: authenticate success, global_id 44107 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 1 -- 172.21.15.5:0/2353201140 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808011990 con 0x7f88140a1eb0 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 1 -- 172.21.15.5:0/2353201140 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808011d40 con 0x7f88140a1eb0 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 4 set_mon_vals no callback set 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8811ffb700 10 monclient: dump: 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 10 monclient: shutdown 2024-09-16T10:53:53.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- 172.21.15.5:0/2353201140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88140a1eb0 msgr2=0x7f88140a22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 --2- 172.21.15.5:0/2353201140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88140a1eb0 0x7f88140a22b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8808009850 tx=0x7f8808009b60 comp rx=0 tx=0).stop 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- 172.21.15.5:0/2353201140 shutdown_connections 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 --2- 172.21.15.5:0/2353201140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88140a1eb0 0x7f88140a22b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 --2- 172.21.15.5:0/2353201140 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1ad0 0x7f88140a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- 172.21.15.5:0/2353201140 >> 172.21.15.5:0/2353201140 conn(0x7f88140a68a0 msgr2=0x7f88140a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- 172.21.15.5:0/2353201140 shutdown_connections 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.258+0000 7f8813fff700 1 -- 172.21.15.5:0/2353201140 wait complete. 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: monmap: 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.275 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 librados: starting msgr at 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 librados: starting objecter 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 Processor -- start 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- start start 2024-09-16T10:53:53.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 librados: setting wanted keys 2024-09-16T10:53:53.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 librados: calling monclient init 2024-09-16T10:53:53.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: init 2024-09-16T10:53:53.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 0x7f88141388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-a con 0x7f88140a1ad0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 0x7f8814122060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-c con 0x7f88140a1eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814139c90 0x7f88141225a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8814139c90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88141248d0 con 0x7f88140a1ad0 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140a5930 con 0x7f8814139c90 2024-09-16T10:53:53.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88140a5470 con 0x7f88140a1eb0 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient(hunting): authenticate will time out at 1091.841804s 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 0x7f8814122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 0x7f88141388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 0x7f8814122060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39122/0 (socket says 172.21.15.5:39122) 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 1 -- 172.21.15.5:0/2770645266 learned_addr learned my addr 172.21.15.5:0/2770645266 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): get_auth_request con 0x7f88140a1eb0 auth_method 0 2024-09-16T10:53:53.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): get_auth_request con 0x7f88140a1ad0 auth_method 0 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814139c90 0x7f88141225a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 10 monclient(hunting): get_auth_request con 0x7f8814139c90 auth_method 0 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient(hunting): handle_auth_done global_id 28464 payload 824 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 msgr2=0x7f8814122060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 0x7f8814122060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814139c90 msgr2=0x7f88141225a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814139c90 0x7f88141225a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 monclient: found mon.noname-a 2024-09-16T10:53:53.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080094b0 con 0x7f88140a1ad0 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88140a1eb0 0x7f8814122060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f88137fe700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814139c90 0x7f88141225a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8812ffd700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 0x7f88141388d0 secure :-1 s=READY pgs=6744 cs=0 l=1 rev1=1 crypto rx=0x7f880400e820 tx=0x7f880400eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 client.?.objecter ms_handle_connect 0x7f88140a1ad0 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804013680 con 0x7f88140a1ad0 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: dump: 2024-09-16T10:53:53.286 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 5 monclient: authenticate success, global_id 28464 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88040137e0 con 0x7f88140a1ad0 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _renew_subs 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881409d2b0 con 0x7f88140a1ad0 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804013b90 con 0x7f88140a1ad0 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 client.28464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8810ff9700 4 set_mon_vals no callback set 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f87fbfff700 10 monclient: dump: 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.288 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _renew_subs 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88141245c0 con 0x7f88140a1ad0 2024-09-16T10:53:53.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.262+0000 7f8813fff700 1 librados: init done 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f8813fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f8813fff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 0x7f881406e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814067380 con 0x7f8814122ae0 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f8804013cf0 con 0x7f88140a1ad0 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 0x7f881406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient: get_auth_request con 0x7f8814122ae0 auth_method 0 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 --2- 172.21.15.5:0/2770645266 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8800036f60 0x7f8800039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4916+0+0 (secure 0 0 0) 0x7f8804047b90 con 0x7f88140a1ad0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 client.28464.objecter ms_dispatch 0x7f8814000fd0 osd_map(168..168 src has 1..168) v4 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 3 client.28464.objecter handle_osd_map got epochs [168,168] > 0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 3 client.28464.objecter handle_osd_map decoding full epoch 168 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 20 client.28464.objecter dump_active .. 0 homeless 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88137fe700 1 --2- 172.21.15.5:0/2770645266 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8800036f60 0x7f8800039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88137fe700 10 monclient: get_auth_request con 0x7f8800036f60 auth_method 0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 10 monclient(hunting): handle_auth_done global_id 44116 payload 824 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f881409cca0 con 0x7f8814122ae0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 client.28464.objecter ms_handle_connect 0x7f8814122ae0 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 client.28464.objecter resend_mon_ops 2024-09-16T10:53:53.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 0x7f881406e330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c00 tx=0x7f8808004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 client.28464.objecter ms_handle_connect 0x7f8800036f60 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f88137fe700 1 --2- 172.21.15.5:0/2770645266 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8800036f60 0x7f8800039420 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7f880c0012e0 tx=0x7f880c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808020070 con 0x7f8814122ae0 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 monclient: dump: 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.292 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f88080036c0 con 0x7f8814122ae0 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 msgr2=0x7f881406e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 0x7f881406e330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8808000c00 tx=0x7f8808004770 comp rx=0 tx=0).stop 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 msgr2=0x7f881406e330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:53.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.266+0000 7f87fbfff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8814122ae0 0x7f881406e330 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f8813fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f8813fff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f8813fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8814125930 con 0x7f881406e870 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f88127fc700 10 monclient: get_auth_request con 0x7f881406e870 auth_method 0 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f88127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f88127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.278+0000 7f88127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 10 monclient(hunting): handle_auth_done global_id 44119 payload 824 2024-09-16T10:53:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 1 -- 172.21.15.5:0/2770645266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8814125730 con 0x7f881406e870 2024-09-16T10:53:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 msgr2=0x7f881406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:53:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 msgr2=0x7f881406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:53:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f8814123870 tx=0x7f8808011000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:53:53.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.282+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:53.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: get_auth_request con 0x7f881406e870 auth_method 0 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: _init_auth method 2 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: _init_auth already have auth, reseting 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: handle_auth_reply_more payload 9 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T10:53:53.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 10 monclient: handle_auth_done global_id 44119 payload 293 2024-09-16T10:53:53.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 client.28464.objecter ms_handle_connect 0x7f881406e870 2024-09-16T10:53:53.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 client.28464.objecter resend_mon_ops 2024-09-16T10:53:53.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88127fc700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8808032000 tx=0x7f8808023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808003e80 con 0x7f881406e870 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 monclient: dump: 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.492 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f880803c070 con 0x7f881406e870 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 msgr2=0x7f881406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8808032000 tx=0x7f8808023000 comp rx=0 tx=0).stop 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 msgr2=0x7f881406ec80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f87fbfff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881406e870 0x7f881406ec80 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8808032000 tx=0x7f8808023000 comp rx=0 tx=0).stop 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:53.493 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.494 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.495 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:53.496 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:53.497 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.498 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.499 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.500 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:53.501 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:53.503 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:53.504 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:53.505 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:53.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 10 librados: watch_flush enter 2024-09-16T10:53:53.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 10 librados: watch_flush exit 2024-09-16T10:53:53.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 20 client.28464.objecter shutdown clearing up homeless session... 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 10 client.28464.objecter successfully canceled tick 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8800036f60 msgr2=0x7f8800039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 --2- 172.21.15.5:0/2770645266 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8800036f60 0x7f8800039420 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7f880c0012e0 tx=0x7f880c009040 comp rx=0 tx=0).stop 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 10 monclient: shutdown 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 msgr2=0x7f88141388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 0x7f88141388d0 secure :-1 s=READY pgs=6744 cs=0 l=1 rev1=1 crypto rx=0x7f880400e820 tx=0x7f880400eb30 comp rx=0 tx=0).stop 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f88137fe700 1 -- 172.21.15.5:0/2770645266 reap_dead start 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 shutdown_connections 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 --2- 172.21.15.5:0/2770645266 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88140a1ad0 0x7f88141388d0 unknown :-1 s=CLOSED pgs=6744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.486+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 >> 172.21.15.5:0/2770645266 conn(0x7f88140a68a0 msgr2=0x7f88140a41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.490+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 shutdown_connections 2024-09-16T10:53:53.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.490+0000 7f8813fff700 1 -- 172.21.15.5:0/2770645266 wait complete. 2024-09-16T10:53:53.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.490+0000 7f8813fff700 1 librados: shutdown 2024-09-16T10:53:53.510 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:53.511 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:53:53.511 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:53:53.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: get_monmap_and_config 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: monmap: 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:53.630597+0000 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:53.630597+0000 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:53.631 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:53.632 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 Processor -- start 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 -- start start 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: init 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c09b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 0x7f7d4c09c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-c con 0x7f7d4c09bd00 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d4c1326b0 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c09fd30 con 0x7f7d4c1326b0 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c09e350 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c09ca20 con 0x7f7d4c09bd00 2024-09-16T10:53:53.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d530c4700 10 monclient(hunting): authenticate will time out at 1092.206988s 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60836/0 (socket says 172.21.15.5:60836) 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 -- 172.21.15.5:0/1003432091 learned_addr learned my addr 172.21.15.5:0/1003432091 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): get_auth_request con 0x7f7d4c09b3e0 auth_method 0 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d528c3700 1 --1- 172.21.15.5:0/1003432091 >> v1:172.21.15.5:6789/0 conn(0x7f7d4c1326b0 0x7f7d4c132b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59166/0 (socket says 172.21.15.5:59166) 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d518c1700 1 --2- 172.21.15.5:0/1003432091 >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 0x7f7d4c09c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient(hunting): handle_auth_done global_id 28467 payload 274 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 -- 172.21.15.5:0/1003432091 >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 msgr2=0x7f7d4c09c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 --2- 172.21.15.5:0/1003432091 >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 0x7f7d4c09c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 -- 172.21.15.5:0/1003432091 >> v1:172.21.15.5:6789/0 conn(0x7f7d4c1326b0 legacy=0x7f7d4c132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 monclient: found mon.noname-a 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 -- 172.21.15.5:0/1003432091 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c09cd00 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d518c1700 10 monclient: get_auth_request con 0x7f7d4c09bd00 auth_method 0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d518c1700 1 --2- 172.21.15.5:0/1003432091 >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 0x7f7d4c09c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.626+0000 7f7d520c2700 1 --2- 172.21.15.5:0/1003432091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 secure :-1 s=READY pgs=6746 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009910 tx=0x7f7d48009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8337ee598d250483 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 1 -- 172.21.15.5:0/1003432091 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4800ba20 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: dump: 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.640 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 5 monclient: authenticate success, global_id 28467 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 1 -- 172.21.15.5:0/1003432091 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4800bb80 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 4 set_mon_vals no callback set 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 1 -- 172.21.15.5:0/1003432091 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48009e60 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:53.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d510c0700 10 monclient: dump: 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: shutdown 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- 172.21.15.5:0/1003432091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 msgr2=0x7f7d4c09b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- 172.21.15.5:0/1003432091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 secure :-1 s=READY pgs=6746 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009910 tx=0x7f7d48009c20 comp rx=0 tx=0).stop 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- 172.21.15.5:0/1003432091 shutdown_connections 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- 172.21.15.5:0/1003432091 >> v2:172.21.15.143:3300/0 conn(0x7f7d4c09bd00 0x7f7d4c09c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- 172.21.15.5:0/1003432091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c09b7c0 unknown :-1 s=CLOSED pgs=6746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- 172.21.15.5:0/1003432091 >> 172.21.15.5:0/1003432091 conn(0x7f7d4c0a45d0 msgr2=0x7f7d4c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- 172.21.15.5:0/1003432091 shutdown_connections 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- 172.21.15.5:0/1003432091 wait complete. 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: build_initial_monmap 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: monmap: 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:53.643 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 librados: starting msgr at 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 librados: starting objecter 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 Processor -- start 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- start start 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 librados: setting wanted keys 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 librados: calling monclient init 2024-09-16T10:53:53.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: init 2024-09-16T10:53:53.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c09b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 0x7f7d4c131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-c con 0x7f7d4c09bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c1326b0 0x7f7d4c13a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient(hunting): picked mon.noname-b con 0x7f7d4c1326b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.630+0000 7f7d530c4700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c11bb70 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c11be50 con 0x7f7d4c1326b0 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c11a650 con 0x7f7d4c09bd00 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient(hunting): authenticate will time out at 1092.212117s 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 0x7f7d4c131240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60842/0 (socket says 172.21.15.5:60842) 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 -- 172.21.15.5:0/3704654149 learned_addr learned my addr 172.21.15.5:0/3704654149 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): get_auth_request con 0x7f7d4c09b3e0 auth_method 0 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): get_auth_request con 0x7f7d4c09bd00 auth_method 0 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c1326b0 0x7f7d4c13a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): get_auth_request con 0x7f7d4c1326b0 auth_method 0 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:53.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d528c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient(hunting): handle_auth_done global_id 28473 payload 824 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 -- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 msgr2=0x7f7d4c131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 0x7f7d4c131240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 -- 172.21.15.5:0/3704654149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c1326b0 msgr2=0x7f7d4c13a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c1326b0 0x7f7d4c13a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient: _finish_hunting 0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 monclient: found mon.noname-a 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 -- 172.21.15.5:0/3704654149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48009570 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 0x7f7d4c131240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 client.?.objecter ms_handle_connect 0x7f7d4c09b3e0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d520c2700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 secure :-1 s=READY pgs=6747 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c075050 tx=0x7f7d48004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48004540 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: dump: 2024-09-16T10:53:53.658 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: _finish_auth 0 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: _check_auth_tickets 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:53.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d480036c0 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 5 monclient: authenticate success, global_id 28473 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _renew_subs 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c0708c0 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48003a50 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 monclient: dump: 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:53.660 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d508bf700 4 set_mon_vals no callback set 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 client.28473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _renew_subs 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c11b890 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 librados: init done 2024-09-16T10:53:53.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7d48021070 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 --2- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 0x7f7d30039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:53.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 4916+0+0 (secure 0 0 0) 0x7f7d48049060 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 10 client.28473.objecter ms_dispatch 0x7f7d4c000fd0 osd_map(168..168 src has 1..168) v4 2024-09-16T10:53:53.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 3 client.28473.objecter handle_osd_map got epochs [168,168] > 0 2024-09-16T10:53:53.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 3 client.28473.objecter handle_osd_map decoding full epoch 168 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d3effd700 20 client.28473.objecter dump_active .. 0 homeless 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c09b7c0 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 1 --2- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 0x7f7d30039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:53.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.634+0000 7f7d518c1700 10 monclient: get_auth_request con 0x7f7d30036f60 auth_method 0 2024-09-16T10:53:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.638+0000 7f7d518c1700 1 --2- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 0x7f7d30039420 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0a2520 tx=0x7f7d40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.638+0000 7f7d3effd700 10 client.28473.objecter ms_handle_connect 0x7f7d30036f60 2024-09-16T10:53:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.642+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7d48015020 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.642+0000 7f7d3effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:53.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.642+0000 7f7d3effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:53.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.890+0000 7f7d530c4700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:53:53.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.890+0000 7f7d530c4700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:53:53.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.890+0000 7f7d530c4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:53.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.890+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7d4c09dff0 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7d3effd700 1 -- 172.21.15.5:0/3704654149 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7d48032070 con 0x7f7d4c09b3e0 2024-09-16T10:53:53.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7d3effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:53:53.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7d3effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=3232530936,config=692619927,health=1621920962,kv=564032378,logm=1301517489}) 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=10,logm=66} crc {auth=4050171740,config=692619927,health=1970776258,kv=564032378,logm=224173095}) 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=3232530936,config=692619927,health=1621920962,kv=564032378,logm=1301517489}) 2024-09-16T10:53:53.900 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.894+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=1380972369,config=692619927,health=1941085706,kv=9945813,logm=4185694944}) 2024-09-16T10:53:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 10 librados: watch_flush enter 2024-09-16T10:53:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 10 librados: watch_flush exit 2024-09-16T10:53:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 20 client.28473.objecter shutdown clearing up homeless session... 2024-09-16T10:53:53.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 10 client.28473.objecter successfully canceled tick 2024-09-16T10:53:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 msgr2=0x7f7d30039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 0x7f7d30039420 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0a2520 tx=0x7f7d40008040 comp rx=0 tx=0).stop 2024-09-16T10:53:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 10 monclient: shutdown 2024-09-16T10:53:53.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 msgr2=0x7f7d4c130d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 secure :-1 s=READY pgs=6747 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c075050 tx=0x7f7d48004770 comp rx=0 tx=0).stop 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 shutdown_connections 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d4c1326b0 0x7f7d4c13a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7d4c09bd00 0x7f7d4c131240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7d30036f60 0x7f7d30039420 unknown :-1 s=CLOSED pgs=3740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 --2- 172.21.15.5:0/3704654149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d4c09b3e0 0x7f7d4c130d00 unknown :-1 s=CLOSED pgs=6747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:53.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 >> 172.21.15.5:0/3704654149 conn(0x7f7d4c0a45d0 msgr2=0x7f7d4c0a2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 shutdown_connections 2024-09-16T10:53:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 -- 172.21.15.5:0/3704654149 wait complete. 2024-09-16T10:53:53.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:53.898+0000 7f7d530c4700 1 librados: shutdown 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843415983}) 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695719120}) 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843415983}) 2024-09-16T10:53:53.907 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.902+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084113946}) 2024-09-16T10:53:53.909 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.910 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420589898}) 2024-09-16T10:53:53.910 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550684388}) 2024-09-16T10:53:53.910 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.910 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420589898}) 2024-09-16T10:53:53.910 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2718389361}) 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405088073}) 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=50933462}) 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405088073}) 2024-09-16T10:53:53.911 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567446678}) 2024-09-16T10:53:53.912 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.912 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239700794}) 2024-09-16T10:53:53.913 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2334220852}) 2024-09-16T10:53:53.913 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.913 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239700794}) 2024-09-16T10:53:53.913 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.906+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830923101}) 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969593805}) 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=407620732}) 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969593805}) 2024-09-16T10:53:53.914 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171346364}) 2024-09-16T10:53:53.915 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.915 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726590009}) 2024-09-16T10:53:53.916 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473556770}) 2024-09-16T10:53:53.916 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.916 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726590009}) 2024-09-16T10:53:53.916 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.910+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4120530280}) 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649388555}) 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458951150}) 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649388555}) 2024-09-16T10:53:53.917 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276139279}) 2024-09-16T10:53:53.918 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.918 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053242751}) 2024-09-16T10:53:53.919 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927430106}) 2024-09-16T10:53:53.919 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.919 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053242751}) 2024-09-16T10:53:53.919 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991937834}) 2024-09-16T10:53:53.920 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.920 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346633942}) 2024-09-16T10:53:53.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989444005}) 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346633942}) 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.914+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529247285}) 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734761782}) 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2428894231}) 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.922 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734761782}) 2024-09-16T10:53:53.923 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866579951}) 2024-09-16T10:53:53.923 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.923 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76095250}) 2024-09-16T10:53:53.924 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489801088}) 2024-09-16T10:53:53.924 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.924 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76095250}) 2024-09-16T10:53:53.924 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691994446}) 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.918+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133644928}) 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773568395}) 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133644928}) 2024-09-16T10:53:53.925 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485074307}) 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683056791}) 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1609071084}) 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1683056791}) 2024-09-16T10:53:53.926 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206338519}) 2024-09-16T10:53:53.927 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.927 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199357470}) 2024-09-16T10:53:53.927 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=646853115}) 2024-09-16T10:53:53.928 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.928 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199357470}) 2024-09-16T10:53:53.928 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884928772}) 2024-09-16T10:53:53.929 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.929 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531201527}) 2024-09-16T10:53:53.929 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2558176953}) 2024-09-16T10:53:53.929 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.929 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531201527}) 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.922+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999722977}) 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685211833}) 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162738402}) 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685211833}) 2024-09-16T10:53:53.930 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061895129}) 2024-09-16T10:53:53.931 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.931 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214067242}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808159444}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214067242}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1017783026}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820506797}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912902781}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820506797}) 2024-09-16T10:53:53.932 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3490427623}) 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366790217}) 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093460120}) 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366790217}) 2024-09-16T10:53:53.933 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.926+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706049758}) 2024-09-16T10:53:53.934 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.934 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=12} crc {logm=28690314,mdsmap=1531151339,mgr=3015988024,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=4180137367,mon_config_key=4284816287,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=1174296766,osd_snap=242479191,osdmap=692346171}) 2024-09-16T10:53:53.934 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=11} crc {logm=122675835,mdsmap=1531151339,mgr=2084859950,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=3443526413,mon_config_key=3273021364,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=1174296766,osd_snap=1078492848,osdmap=3721718495}) 2024-09-16T10:53:53.935 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.935 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=12} crc {logm=28690314,mdsmap=1531151339,mgr=3015988024,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=4180137367,mon_config_key=4284816287,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=1174296766,osd_snap=242479191,osdmap=692346171}) 2024-09-16T10:53:53.935 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=11} crc {logm=3192198671,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=83670491,mon_config_key=3273021363,monmap=3140514009,osd_metadata=3752038628,osd_pg_creating=1174296766,osd_snap=3741025822,osdmap=3721718495}) 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2972483}) 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1393374392}) 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2972483}) 2024-09-16T10:53:53.936 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.930+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=678771398}) 2024-09-16T10:53:53.938 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.938 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1464518465}) 2024-09-16T10:53:53.938 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2001815395}) 2024-09-16T10:53:53.938 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.938 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1464518465}) 2024-09-16T10:53:53.939 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.934+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2105876583}) 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.938+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.938+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=1049616414}) 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.938+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=1810727773}) 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.938+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.942+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=1049616414}) 2024-09-16T10:53:53.945 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:53:53.942+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=44} crc {osdmap=3368986193}) 2024-09-16T10:53:54.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:53:54.923 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: get_monmap_and_config 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: monmap: 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:55.041754+0000 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:55.041754+0000 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:55.043 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 Processor -- start 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 -- start start 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: init 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a2eb0 0x7f0d600a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-b con 0x7f0d600a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0d600a4280 0x7f0d600a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-c con 0x7f0d600a4280 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0d600a3d10 0x7f0d60135c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-a con 0x7f0d600a3d10 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d600a1e50 con 0x7f0d600a3d10 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6009fdc0 con 0x7f0d600a4280 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6009fae0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d654f5700 10 monclient(hunting): authenticate will time out at 1093.618438s 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0d600a3d10 0x7f0d60135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a2eb0 0x7f0d600a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0d600a3d10 0x7f0d60135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60856/0 (socket says 172.21.15.5:60856) 2024-09-16T10:53:55.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 -- 172.21.15.5:0/322316039 learned_addr learned my addr 172.21.15.5:0/322316039 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600a3d10 auth_method 0 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): get_auth_request con 0x7f0d600a2eb0 auth_method 0 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 1 --2- 172.21.15.5:0/322316039 >> v2:172.21.15.143:3300/0 conn(0x7f0d600a4280 0x7f0d600a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): get_auth_request con 0x7f0d600a4280 auth_method 0 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_done global_id 28476 payload 274 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 -- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a2eb0 msgr2=0x7f0d600a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 --2- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a2eb0 0x7f0d600a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 -- 172.21.15.5:0/322316039 >> v2:172.21.15.143:3300/0 conn(0x7f0d600a4280 msgr2=0x7f0d600a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 --2- 172.21.15.5:0/322316039 >> v2:172.21.15.143:3300/0 conn(0x7f0d600a4280 0x7f0d600a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 monclient: found mon.noname-a 2024-09-16T10:53:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 -- 172.21.15.5:0/322316039 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6009efb0 con 0x7f0d600a3d10 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5f7fe700 1 --2- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d60135c70 secure :-1 s=READY pgs=6752 cs=0 l=1 rev1=1 crypto rx=0x7f0d6006be60 tx=0x7f0d58009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3127ae7983c15157 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5dffb700 1 -- 172.21.15.5:0/322316039 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58011830 con 0x7f0d600a3d10 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.038+0000 7f0d5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: dump: 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 5 monclient: authenticate success, global_id 28476 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 1 -- 172.21.15.5:0/322316039 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d58011990 con 0x7f0d600a3d10 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 4 set_mon_vals no callback set 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 1 -- 172.21.15.5:0/322316039 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58011d20 con 0x7f0d600a3d10 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5dffb700 10 monclient: dump: 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: shutdown 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 msgr2=0x7f0d60135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d60135c70 secure :-1 s=READY pgs=6752 cs=0 l=1 rev1=1 crypto rx=0x7f0d6006be60 tx=0x7f0d58009bf0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- 172.21.15.5:0/322316039 shutdown_connections 2024-09-16T10:53:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d60135c70 unknown :-1 s=CLOSED pgs=6752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- 172.21.15.5:0/322316039 >> v2:172.21.15.143:3300/0 conn(0x7f0d600a4280 0x7f0d600a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- 172.21.15.5:0/322316039 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a2eb0 0x7f0d600a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- 172.21.15.5:0/322316039 >> 172.21.15.5:0/322316039 conn(0x7f0d600a4fb0 msgr2=0x7f0d600a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- 172.21.15.5:0/322316039 shutdown_connections 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- 172.21.15.5:0/322316039 wait complete. 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: monmap: 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 librados: starting msgr at 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 librados: starting objecter 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 Processor -- start 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- start start 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 librados: setting wanted keys 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 librados: calling monclient init 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: init 2024-09-16T10:53:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 0x7f0d60134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-b con 0x7f0d600a2eb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-a con 0x7f0d600a3d10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a4280 0x7f0d60122950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): picked mon.noname-c con 0x7f0d600a4280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d601cb420 con 0x7f0d600a3d10 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600a6ca0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d60124a60 con 0x7f0d600a4280 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d654f5700 10 monclient(hunting): authenticate will time out at 1093.622422s 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60870/0 (socket says 172.21.15.5:60870) 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 1 -- 172.21.15.5:0/398494276 learned_addr learned my addr 172.21.15.5:0/398494276 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): get_auth_request con 0x7f0d600a3d10 auth_method 0 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a4280 0x7f0d60122950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 0x7f0d60134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600a4280 auth_method 0 2024-09-16T10:53:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): get_auth_request con 0x7f0d600a2eb0 auth_method 0 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient(hunting): handle_auth_done global_id 44128 payload 824 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 -- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a4280 msgr2=0x7f0d60122950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a4280 0x7f0d60122950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 -- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 msgr2=0x7f0d601353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 monclient: found mon.noname-b 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 -- 172.21.15.5:0/398494276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d58009570 con 0x7f0d600a2eb0 2024-09-16T10:53:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5e7fc700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d5effd700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 0x7f0d60134e90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400e930 tx=0x7f0d5400ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d477fe700 10 client.?.objecter ms_handle_connect 0x7f0d600a2eb0 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.042+0000 7f0d477fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54013680 con 0x7f0d600a2eb0 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: dump: 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.066 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 5 monclient: authenticate success, global_id 44128 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d540137e0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _renew_subs 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6009f6a0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54013b50 con 0x7f0d600a2eb0 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 monclient: dump: 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.068 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 client.44128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _renew_subs 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d601268b0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d5d7fa700 4 set_mon_vals no callback set 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 1 librados: init done 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0d54013cb0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 --2- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 0x7f0d4c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6039+0+0 (secure 0 0 0) 0x7f0d540483d0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 client.44128.objecter ms_dispatch 0x7f0d60000fd0 osd_map(170..170 src has 1..170) v4 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 3 client.44128.objecter handle_osd_map got epochs [170,170] > 0 2024-09-16T10:53:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 3 client.44128.objecter handle_osd_map decoding full epoch 170 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 20 client.44128.objecter dump_active .. 0 homeless 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d5e7fc700 1 --2- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 0x7f0d4c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d5e7fc700 10 monclient: get_auth_request con 0x7f0d4c036f60 auth_method 0 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60136130 con 0x7f0d600a2eb0 2024-09-16T10:53:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d477fe700 10 client.44128.objecter ms_handle_connect 0x7f0d4c036f60 2024-09-16T10:53:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.046+0000 7f0d5e7fc700 1 --2- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 0x7f0d4c039420 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7f0d60124150 tx=0x7f0d5000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.058+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0d5401adc0 con 0x7f0d600a2eb0 2024-09-16T10:53:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.058+0000 7f0d477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.058+0000 7f0d477fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:55.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d654f5700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:53:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d654f5700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d654f5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d60135c70 con 0x7f0d600a2eb0 2024-09-16T10:53:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d477fe700 1 -- 172.21.15.5:0/398494276 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f0d5403b610 con 0x7f0d600a2eb0 2024-09-16T10:53:55.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:55.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.294+0000 7f0d477fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:55.301 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:53:55.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 10 librados: watch_flush enter 2024-09-16T10:53:55.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 10 librados: watch_flush exit 2024-09-16T10:53:55.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 20 client.44128.objecter shutdown clearing up homeless session... 2024-09-16T10:53:55.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 10 client.44128.objecter successfully canceled tick 2024-09-16T10:53:55.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 msgr2=0x7f0d4c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 0x7f0d4c039420 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7f0d60124150 tx=0x7f0d5000b040 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 10 monclient: shutdown 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 msgr2=0x7f0d60134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 0x7f0d60134e90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400e930 tx=0x7f0d5400ec40 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 shutdown_connections 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0d600a4280 0x7f0d60122950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0d4c036f60 0x7f0d4c039420 unknown :-1 s=CLOSED pgs=3741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0d600a3d10 0x7f0d601353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 --2- 172.21.15.5:0/398494276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d600a2eb0 0x7f0d60134e90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 >> 172.21.15.5:0/398494276 conn(0x7f0d600a4fb0 msgr2=0x7f0d600a7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 shutdown_connections 2024-09-16T10:53:55.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 -- 172.21.15.5:0/398494276 wait complete. 2024-09-16T10:53:55.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.298+0000 7f0d654f5700 1 librados: shutdown 2024-09-16T10:53:55.316 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:53:55.316 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:53:55.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:53:55.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:53:55.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 10 monclient: get_monmap_and_config 2024-09-16T10:53:55.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 10 monclient: monmap: 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:55.426280+0000 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:55.426280+0000 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:55.427 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:55.428 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:55.428 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:55.428 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:55.428 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 1 Processor -- start 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 1 -- start start 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 10 monclient: init 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.422+0000 7f748bfff700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f748c0a1dc0 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:55.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 0x7f748c137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f748c072450 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f748c070e40 0x7f748c139b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f748c070e40 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c0a21a0 con 0x7f748c0a1dc0 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c06d9e0 con 0x7f748c070e40 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c132c80 con 0x7f748c072450 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient(hunting): authenticate will time out at 1094.004229s 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748affd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f748c0a1dc0 0x7f748c070730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59182/0 (socket says 172.21.15.5:59182) 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748affd700 1 -- 172.21.15.5:0/1558228992 learned_addr learned my addr 172.21.15.5:0/1558228992 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:53:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 --2- 172.21.15.5:0/1558228992 >> v2:172.21.15.5:3300/0 conn(0x7f748c070e40 0x7f748c139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): get_auth_request con 0x7f748c070e40 auth_method 0 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f7483fff700 1 --2- 172.21.15.5:0/1558228992 >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 0x7f748c137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient(hunting): handle_auth_done global_id 28482 payload 274 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 -- 172.21.15.5:0/1558228992 >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 msgr2=0x7f748c137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 --2- 172.21.15.5:0/1558228992 >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 0x7f748c137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 -- 172.21.15.5:0/1558228992 >> v1:172.21.15.5:6789/0 conn(0x7f748c0a1dc0 legacy=0x7f748c070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 monclient: found mon.noname-a 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 -- 172.21.15.5:0/1558228992 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c06e960 con 0x7f748c070e40 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 1 -- 172.21.15.5:0/1558228992 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461581093 0 0) 0x7f748c0a21a0 con 0x7f748c0a1dc0 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f7483fff700 10 monclient: get_auth_request con 0x7f748c072450 auth_method 0 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f7483fff700 1 --2- 172.21.15.5:0/1558228992 >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 0x7f748c137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748b7fe700 1 --2- 172.21.15.5:0/1558228992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c139b50 secure :-1 s=READY pgs=6754 cs=0 l=1 rev1=1 crypto rx=0x7f748c06e5d0 tx=0x7f748400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100aeb30b9821fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 1 -- 172.21.15.5:0/1558228992 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748400e070 con 0x7f748c070e40 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:55.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: dump: 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 5 monclient: authenticate success, global_id 28482 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 1 -- 172.21.15.5:0/1558228992 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748400bbd0 con 0x7f748c070e40 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 4 set_mon_vals no callback set 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 1 -- 172.21.15.5:0/1558228992 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484012430 con 0x7f748c070e40 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748a7fc700 10 monclient: dump: 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.438 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient: shutdown 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- 172.21.15.5:0/1558228992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 msgr2=0x7f748c139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 --2- 172.21.15.5:0/1558228992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c139b50 secure :-1 s=READY pgs=6754 cs=0 l=1 rev1=1 crypto rx=0x7f748c06e5d0 tx=0x7f748400b330 comp rx=0 tx=0).stop 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- 172.21.15.5:0/1558228992 shutdown_connections 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 --2- 172.21.15.5:0/1558228992 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c139b50 unknown :-1 s=CLOSED pgs=6754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 --2- 172.21.15.5:0/1558228992 >> v2:172.21.15.143:3300/0 conn(0x7f748c072450 0x7f748c137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- 172.21.15.5:0/1558228992 >> 172.21.15.5:0/1558228992 conn(0x7f748c0a12a0 msgr2=0x7f748c0a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- 172.21.15.5:0/1558228992 shutdown_connections 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 -- 172.21.15.5:0/1558228992 wait complete. 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 10 monclient: monmap: 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:55.440 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 librados: starting msgr at 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.426+0000 7f748bfff700 1 librados: starting objecter 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 Processor -- start 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- start start 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 librados: setting wanted keys 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 librados: calling monclient init 2024-09-16T10:53:55.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: init 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c1396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f748c070e40 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f748c072450 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f748c0a1dc0 0x7f748c131970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f748c0a1dc0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c138f40 con 0x7f748c070e40 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c136be0 con 0x7f748c0a1dc0 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c09f820 con 0x7f748c072450 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): authenticate will time out at 1094.009099s 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c1396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c1396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60880/0 (socket says 172.21.15.5:60880) 2024-09-16T10:53:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39186/0 (socket says 172.21.15.5:39186) 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 1 -- 172.21.15.5:0/2619333107 learned_addr learned my addr 172.21.15.5:0/2619333107 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): get_auth_request con 0x7f748c070e40 auth_method 0 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): get_auth_request con 0x7f748c072450 auth_method 0 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f748c0a1dc0 0x7f748c131970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 10 monclient(hunting): get_auth_request con 0x7f748c0a1dc0 auth_method 0 2024-09-16T10:53:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient(hunting): handle_auth_done global_id 22589 payload 824 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f748c0a1dc0 msgr2=0x7f748c131970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f748c0a1dc0 0x7f748c131970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 msgr2=0x7f748c1396d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c070e40 0x7f748c1396d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 monclient: found mon.noname-c 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484009a70 con 0x7f748c072450 2024-09-16T10:53:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748b7fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f748c0a1dc0 0x7f748c131970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f7483fff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 secure :-1 s=READY pgs=4628 cs=0 l=1 rev1=1 crypto rx=0x7f747400c7f0 tx=0x7f747400cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 client.?.objecter ms_handle_connect 0x7f748c072450 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474011800 con 0x7f748c072450 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: dump: 2024-09-16T10:53:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 5 monclient: authenticate success, global_id 22589 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _renew_subs 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c0a40a0 con 0x7f748c072450 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7474011960 con 0x7f748c072450 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474011cd0 con 0x7f748c072450 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74837fe700 10 monclient: dump: 2024-09-16T10:53:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f74897fa700 4 set_mon_vals no callback set 2024-09-16T10:53:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 client.22589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _renew_subs 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c131eb0 con 0x7f748c072450 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 librados: init done 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 0x7f748c135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c074c30 con 0x7f748c1386a0 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 0x7f748c135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient: get_auth_request con 0x7f748c1386a0 auth_method 0 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.430+0000 7f748affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 10 monclient(hunting): handle_auth_done global_id 28491 payload 824 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7474022350 con 0x7f748c072450 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748c132930 con 0x7f748c1386a0 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748affd700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 0x7f748c135790 secure :-1 s=READY pgs=6755 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f747c00fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 --2- 172.21.15.5:0/2619333107 >> v2:172.21.15.143:6812/1098205034 conn(0x7f746c036f60 0x7f746c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 client.22589.objecter ms_handle_connect 0x7f748c1386a0 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 client.22589.objecter resend_mon_ops 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6039+0+0 (secure 0 0 0) 0x7f7474048120 con 0x7f748c072450 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 client.22589.objecter ms_dispatch 0x7f748c000fd0 osd_map(170..170 src has 1..170) v4 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 3 client.22589.objecter handle_osd_map got epochs [170,170] > 0 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 3 client.22589.objecter handle_osd_map decoding full epoch 170 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 20 client.22589.objecter dump_active .. 0 homeless 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c005b50 con 0x7f748c1386a0 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 monclient: dump: 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748b7fe700 1 --2- 172.21.15.5:0/2619333107 >> v2:172.21.15.143:6812/1098205034 conn(0x7f746c036f60 0x7f746c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748b7fe700 10 monclient: get_auth_request con 0x7f746c036f60 auth_method 0 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f747c005cb0 con 0x7f748c1386a0 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 msgr2=0x7f748c135790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 0x7f748c135790 secure :-1 s=READY pgs=6755 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f747c00fb00 comp rx=0 tx=0).stop 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 msgr2=0x7f748c135790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c1386a0 0x7f748c135790 secure :-1 s=CLOSED pgs=6755 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f747c00fb00 comp rx=0 tx=0).stop 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f748b7fe700 1 --2- 172.21.15.5:0/2619333107 >> v2:172.21.15.143:6812/1098205034 conn(0x7f746c036f60 0x7f746c039420 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f7484014040 tx=0x7f7484011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.434+0000 7f74837fe700 10 client.22589.objecter ms_handle_connect 0x7f746c036f60 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 0x7f748c02bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c09d210 con 0x7f748c02b6a0 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 0x7f748c02bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient: get_auth_request con 0x7f748c02b6a0 auth_method 0 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 10 monclient(hunting): handle_auth_done global_id 28494 payload 824 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 1 -- 172.21.15.5:0/2619333107 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f748c070730 con 0x7f748c02b6a0 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748affd700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 0x7f748c02bab0 secure :-1 s=READY pgs=6756 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f747c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 client.22589.objecter ms_handle_connect 0x7f748c02b6a0 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 client.22589.objecter resend_mon_ops 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c023070 con 0x7f748c02b6a0 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 monclient: dump: 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.463 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f747c0076f0 con 0x7f748c02b6a0 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 msgr2=0x7f748c02bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 0x7f748c02bab0 secure :-1 s=READY pgs=6756 cs=0 l=1 rev1=1 crypto rx=0x7f747c007fd0 tx=0x7f747c0058a0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 msgr2=0x7f748c02bab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f74837fe700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f748c02b6a0 0x7f748c02bab0 unknown :-1 s=CLOSED pgs=6756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:55.465 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.466 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.467 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.468 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:55.469 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.470 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:55.471 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.472 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.473 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.474 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.475 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.476 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:55.477 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:53:55.478 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.479 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.479 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.479 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:55.479 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:55.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 librados: watch_flush enter 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 librados: watch_flush exit 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 20 client.22589.objecter shutdown clearing up homeless session... 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 client.22589.objecter successfully canceled tick 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 >> v2:172.21.15.143:6812/1098205034 conn(0x7f746c036f60 msgr2=0x7f746c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 --2- 172.21.15.5:0/2619333107 >> v2:172.21.15.143:6812/1098205034 conn(0x7f746c036f60 0x7f746c039420 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f7484014040 tx=0x7f7484011000 comp rx=0 tx=0).stop 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 10 monclient: shutdown 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 msgr2=0x7f748c131430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 secure :-1 s=READY pgs=4628 cs=0 l=1 rev1=1 crypto rx=0x7f747400c7f0 tx=0x7f747400cb00 comp rx=0 tx=0).stop 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748b7fe700 1 -- 172.21.15.5:0/2619333107 reap_dead start 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 shutdown_connections 2024-09-16T10:53:55.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 --2- 172.21.15.5:0/2619333107 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f748c072450 0x7f748c131430 unknown :-1 s=CLOSED pgs=4628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 >> 172.21.15.5:0/2619333107 conn(0x7f748c0a12a0 msgr2=0x7f748c0a2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 shutdown_connections 2024-09-16T10:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 -- 172.21.15.5:0/2619333107 wait complete. 2024-09-16T10:53:55.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.442+0000 7f748bfff700 1 librados: shutdown 2024-09-16T10:53:55.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:55.482 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:53:55.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:53:55.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 10 monclient: get_monmap_and_config 2024-09-16T10:53:55.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 10 monclient: monmap: 2024-09-16T10:53:55.604 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.604 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:55.603929+0000 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:55.603929+0000 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 1 Processor -- start 2024-09-16T10:53:55.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 1 -- start start 2024-09-16T10:53:55.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.598+0000 7fdd5a937700 10 monclient: init 2024-09-16T10:53:55.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T10:53:55.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-b con 0x7fdd5409d6d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 0x7fdd5409de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-a con 0x7fdd5409dab0 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd54133810 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540a0050 con 0x7fdd54133810 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5409fd70 con 0x7fdd5409dab0 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540a0330 con 0x7fdd5409d6d0 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient(hunting): authenticate will time out at 1094.180318s 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39196/0 (socket says 172.21.15.5:39196) 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 0x7fdd5409de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 -- 172.21.15.5:0/3380386802 learned_addr learned my addr 172.21.15.5:0/3380386802 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): get_auth_request con 0x7fdd5409d6d0 auth_method 0 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request con 0x7fdd5409dab0 auth_method 0 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient(hunting): handle_auth_done global_id 22592 payload 274 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 -- 172.21.15.5:0/3380386802 >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 msgr2=0x7fdd5409de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 --2- 172.21.15.5:0/3380386802 >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 0x7fdd5409de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 -- 172.21.15.5:0/3380386802 >> v1:172.21.15.143:6789/0 conn(0x7fdd54133810 legacy=0x7fdd54133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 monclient: found mon.noname-b 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 -- 172.21.15.5:0/3380386802 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5409e3d0 con 0x7fdd5409d6d0 2024-09-16T10:53:55.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/3380386802 >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 0x7fdd5409de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd53fff700 1 --2- 172.21.15.5:0/3380386802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 secure :-1 s=READY pgs=4629 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009910 tx=0x7fdd44009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a129ff3593bed878 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 1 -- 172.21.15.5:0/3380386802 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44011830 con 0x7fdd5409d6d0 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: dump: 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.613 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 5 monclient: authenticate success, global_id 22592 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 1 -- 172.21.15.5:0/3380386802 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44011990 con 0x7fdd5409d6d0 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 4 set_mon_vals no callback set 2024-09-16T10:53:55.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 1 -- 172.21.15.5:0/3380386802 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44011d40 con 0x7fdd5409d6d0 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd537fe700 10 monclient: dump: 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.615 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 10 monclient: shutdown 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- 172.21.15.5:0/3380386802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 msgr2=0x7fdd5409f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 --2- 172.21.15.5:0/3380386802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 secure :-1 s=READY pgs=4629 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009910 tx=0x7fdd44009c20 comp rx=0 tx=0).stop 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- 172.21.15.5:0/3380386802 shutdown_connections 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 --2- 172.21.15.5:0/3380386802 >> v2:172.21.15.5:3300/0 conn(0x7fdd5409dab0 0x7fdd5409de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 --2- 172.21.15.5:0/3380386802 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd5409f3f0 unknown :-1 s=CLOSED pgs=4629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- 172.21.15.5:0/3380386802 >> 172.21.15.5:0/3380386802 conn(0x7fdd540a0b30 msgr2=0x7fdd540a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- 172.21.15.5:0/3380386802 shutdown_connections 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.602+0000 7fdd5a937700 1 -- 172.21.15.5:0/3380386802 wait complete. 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: monmap: 2024-09-16T10:53:55.616 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 librados: starting msgr at 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 librados: starting objecter 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 Processor -- start 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- start start 2024-09-16T10:53:55.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 librados: setting wanted keys 2024-09-16T10:53:55.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 librados: calling monclient init 2024-09-16T10:53:55.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: init 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-c con 0x7fdd5409d6d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5409dab0 0x7fdd541cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-b con 0x7fdd5409dab0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd54133810 0x7fdd5409b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): picked mon.noname-a con 0x7fdd54133810 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540a37b0 con 0x7fdd54133810 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54125f30 con 0x7fdd5409dab0 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5406fa10 con 0x7fdd5409d6d0 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient(hunting): authenticate will time out at 1094.185920s 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39208/0 (socket says 172.21.15.5:39208) 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 -- 172.21.15.5:0/194505082 learned_addr learned my addr 172.21.15.5:0/194505082 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): get_auth_request con 0x7fdd5409d6d0 auth_method 0 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd54133810 0x7fdd5409b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5409dab0 0x7fdd541cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): get_auth_request con 0x7fdd54133810 auth_method 0 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request con 0x7fdd5409dab0 auth_method 0 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd58ed4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient(hunting): handle_auth_done global_id 22595 payload 824 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5409dab0 msgr2=0x7fdd541cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5409dab0 0x7fdd541cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd54133810 msgr2=0x7fdd5409b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd54133810 0x7fdd5409b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 monclient: found mon.noname-c 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0094b0 con 0x7fdd5409d6d0 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd53fff700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 secure :-1 s=READY pgs=4630 cs=0 l=1 rev1=1 crypto rx=0x7fdd44000c00 tx=0x7fdd44004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 client.?.objecter ms_handle_connect 0x7fdd5409d6d0 2024-09-16T10:53:55.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44004510 con 0x7fdd5409d6d0 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: dump: 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.628 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 5 monclient: authenticate success, global_id 22595 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _renew_subs 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd44009570 con 0x7fdd5409d6d0 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 client.22595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _renew_subs 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:53:55.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5409d0f0 con 0x7fdd5409d6d0 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd5a937700 1 librados: init done 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd440036c0 con 0x7fdd5409d6d0 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44003a50 con 0x7fdd5409d6d0 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd52ffd700 4 set_mon_vals no callback set 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.606+0000 7fdd517fa700 10 monclient: dump: 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.630 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd5a937700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd5a937700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 0x7fdd38004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38007d30 con 0x7fdd38002850 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 0x7fdd38004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient: get_auth_request con 0x7fdd38002850 auth_method 0 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fdd44003bb0 con 0x7fdd5409d6d0 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 --2- 172.21.15.5:0/194505082 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdd34036f90 0x7fdd34039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 6039+0+0 (secure 0 0 0) 0x7fdd44021030 con 0x7fdd5409d6d0 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 client.22595.objecter ms_dispatch 0x7fdd54000fd0 osd_map(170..170 src has 1..170) v4 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 3 client.22595.objecter handle_osd_map got epochs [170,170] > 0 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 3 client.22595.objecter handle_osd_map decoding full epoch 170 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_done global_id 22598 payload 824 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 20 client.22595.objecter dump_active .. 0 homeless 2024-09-16T10:53:55.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd38007ef0 con 0x7fdd38002850 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd58ed4700 1 --2- 172.21.15.5:0/194505082 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdd34036f90 0x7fdd34039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd58ed4700 10 monclient: get_auth_request con 0x7fdd34036f90 auth_method 0 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 client.22595.objecter ms_handle_connect 0x7fdd38002850 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 client.22595.objecter resend_mon_ops 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 0x7fdd38004cf0 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 crypto rx=0x7fdd5409ccf0 tx=0x7fdd3c011f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00fb60 con 0x7fdd38002850 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 monclient: dump: 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.633 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fdd3c00fcc0 con 0x7fdd38002850 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 msgr2=0x7fdd38004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 0x7fdd38004cf0 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 crypto rx=0x7fdd5409ccf0 tx=0x7fdd3c011f60 comp rx=0 tx=0).stop 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 msgr2=0x7fdd38004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38002850 0x7fdd38004cf0 secure :-1 s=CLOSED pgs=4631 cs=0 l=1 rev1=1 crypto rx=0x7fdd5409ccf0 tx=0x7fdd3c011f60 comp rx=0 tx=0).stop 2024-09-16T10:53:55.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd517fa700 10 client.22595.objecter ms_handle_connect 0x7fdd34036f90 2024-09-16T10:53:55.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.610+0000 7fdd58ed4700 1 --2- 172.21.15.5:0/194505082 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdd34036f90 0x7fdd34039450 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0012e0 tx=0x7fdd4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 0x7fdd380071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3800a1d0 con 0x7fdd38006d70 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 0x7fdd380071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient: get_auth_request con 0x7fdd38006d70 auth_method 0 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 10 monclient(hunting): handle_auth_done global_id 22601 payload 824 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 1 -- 172.21.15.5:0/194505082 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd3800a390 con 0x7fdd38006d70 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 client.22595.objecter ms_handle_connect 0x7fdd38006d70 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd4b7fe700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 0x7fdd380071e0 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 crypto rx=0x7fdd5409ccf0 tx=0x7fdd3c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 client.22595.objecter resend_mon_ops 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c02c640 con 0x7fdd38006d70 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 monclient: dump: 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.638 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7fdd3c01d400 con 0x7fdd38006d70 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 msgr2=0x7fdd380071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 0x7fdd380071e0 secure :-1 s=READY pgs=4632 cs=0 l=1 rev1=1 crypto rx=0x7fdd5409ccf0 tx=0x7fdd3c004770 comp rx=0 tx=0).stop 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 msgr2=0x7fdd380071e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd517fa700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd38006d70 0x7fdd380071e0 unknown :-1 s=CLOSED pgs=4632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:55.640 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 5, 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.641 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.642 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:55.643 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.644 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.645 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.646 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.647 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.648 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.649 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.650 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.651 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.652 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:55.653 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:55.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 librados: watch_flush enter 2024-09-16T10:53:55.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 librados: watch_flush exit 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 20 client.22595.objecter shutdown clearing up homeless session... 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 client.22595.objecter successfully canceled tick 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdd34036f90 msgr2=0x7fdd34039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 --2- 172.21.15.5:0/194505082 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdd34036f90 0x7fdd34039450 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0012e0 tx=0x7fdd4c009040 comp rx=0 tx=0).stop 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 10 monclient: shutdown 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd58ed4700 1 -- 172.21.15.5:0/194505082 reap_dead start 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 msgr2=0x7fdd541cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 secure :-1 s=READY pgs=4630 cs=0 l=1 rev1=1 crypto rx=0x7fdd44000c00 tx=0x7fdd44004770 comp rx=0 tx=0).stop 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 shutdown_connections 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 --2- 172.21.15.5:0/194505082 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdd5409d6d0 0x7fdd541cacd0 unknown :-1 s=CLOSED pgs=4630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 >> 172.21.15.5:0/194505082 conn(0x7fdd540a0b30 msgr2=0x7fdd5406f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 shutdown_connections 2024-09-16T10:53:55.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 -- 172.21.15.5:0/194505082 wait complete. 2024-09-16T10:53:55.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.622+0000 7fdd5a937700 1 librados: shutdown 2024-09-16T10:53:55.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:55.656 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:53:55.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:53:55.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: get_monmap_and_config 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: monmap: 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:55.778668+0000 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:55.778668+0000 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:55.779 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:55.780 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 Processor -- start 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 -- start start 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: init 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d1809e8e0 0x7f4d1809ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-a con 0x7f4d1809e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d1809d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d1809dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d18070e40 con 0x7f4d1809dfd0 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d181344e0 con 0x7f4d1809d6a0 2024-09-16T10:53:55.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d180a01d0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d1d5ae700 10 monclient(hunting): authenticate will time out at 1094.355032s 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d1809e8e0 0x7f4d1809ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d1809e8e0 0x7f4d1809ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60914/0 (socket says 172.21.15.5:60914) 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 -- 172.21.15.5:0/3207298253 learned_addr learned my addr 172.21.15.5:0/3207298253 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): get_auth_request con 0x7f4d1809e8e0 auth_method 0 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient(hunting): handle_auth_done global_id 28503 payload 274 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 -- 172.21.15.5:0/3207298253 >> v1:172.21.15.143:6789/0 conn(0x7f4d1809d6a0 legacy=0x7f4d181336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 -- 172.21.15.5:0/3207298253 >> v1:172.21.15.5:6789/0 conn(0x7f4d1809dfd0 legacy=0x7f4d18133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 monclient: found mon.noname-a 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.774+0000 7f4d17fff700 1 -- 172.21.15.5:0/3207298253 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1809fef0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d17fff700 1 --2- 172.21.15.5:0/3207298253 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d1809ecc0 secure :-1 s=READY pgs=6758 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009940 tx=0x7f4d08009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e616aceb1d0dfb14 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 1 -- 172.21.15.5:0/3207298253 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08004230 con 0x7f4d1809e8e0 2024-09-16T10:53:55.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: dump: 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 1 -- 172.21.15.5:0/3207298253 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d08009e90 con 0x7f4d1809e8e0 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 5 monclient: authenticate success, global_id 28503 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 1 -- 172.21.15.5:0/3207298253 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d080045b0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 4 set_mon_vals no callback set 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d177fe700 10 monclient: dump: 2024-09-16T10:53:55.788 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: shutdown 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/3207298253 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 msgr2=0x7f4d1809ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/3207298253 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d1809ecc0 secure :-1 s=READY pgs=6758 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009940 tx=0x7f4d08009c50 comp rx=0 tx=0).stop 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/3207298253 shutdown_connections 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/3207298253 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d1809ecc0 unknown :-1 s=CLOSED pgs=6758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/3207298253 >> 172.21.15.5:0/3207298253 conn(0x7f4d180a45d0 msgr2=0x7f4d180a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/3207298253 shutdown_connections 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/3207298253 wait complete. 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: build_initial_monmap 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: monmap: 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:55.790 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 librados: starting msgr at 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 librados: starting objecter 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 Processor -- start 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- start start 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 librados: setting wanted keys 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 librados: calling monclient init 2024-09-16T10:53:55.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: init 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d1809d6a0 0x7f4d181ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-c con 0x7f4d1809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d1809dfd0 0x7f4d18132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-b con 0x7f4d1809dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): picked mon.noname-a con 0x7f4d1809e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18133780 con 0x7f4d1809e8e0 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18122120 con 0x7f4d1809dfd0 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18121190 con 0x7f4d1809d6a0 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.778+0000 7f4d1d5ae700 10 monclient(hunting): authenticate will time out at 1094.359912s 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d1809d6a0 0x7f4d181ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60924/0 (socket says 172.21.15.5:60924) 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d1809d6a0 0x7f4d181ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39236/0 (socket says 172.21.15.5:39236) 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 -- 172.21.15.5:0/2791729314 learned_addr learned my addr 172.21.15.5:0/2791729314 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:55.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): get_auth_request con 0x7f4d1809e8e0 auth_method 0 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): get_auth_request con 0x7f4d1809d6a0 auth_method 0 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d1809dfd0 0x7f4d18132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): get_auth_request con 0x7f4d1809dfd0 auth_method 0 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient(hunting): handle_auth_done global_id 28509 payload 824 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d1809d6a0 msgr2=0x7f4d181ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4d1809d6a0 0x7f4d181ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d1809dfd0 msgr2=0x7f4d18132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d1809dfd0 0x7f4d18132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient: _finish_hunting 0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 monclient: found mon.noname-a 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08009570 con 0x7f4d1809e8e0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1cdad700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 secure :-1 s=READY pgs=6759 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000cbc0 tx=0x7f4d1000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 client.?.objecter ms_handle_connect 0x7f4d1809e8e0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10003c20 con 0x7f4d1809e8e0 2024-09-16T10:53:55.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: dump: 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.803 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: _finish_auth 0 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: _check_auth_tickets 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 5 monclient: authenticate success, global_id 28509 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 monclient: _renew_subs 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d10003d80 con 0x7f4d1809e8e0 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1809b0d0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000aae0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 client.28509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:55.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d16ffd700 4 set_mon_vals no callback set 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: dump: 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.805 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 monclient: _renew_subs 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1809d0b0 con 0x7f4d1809e8e0 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 1 librados: init done 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 0x7f4d181c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18067380 con 0x7f4d181c7000 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 0x7f4d181c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient: get_auth_request con 0x7f4d181c7000 auth_method 0 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4d1000ac40 con 0x7f4d1809e8e0 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 --2- 172.21.15.5:0/2791729314 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4cf8036f60 0x7f4cf8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 10 monclient(hunting): handle_auth_done global_id 44143 payload 824 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d1809f200 con 0x7f4d181c7000 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f4d10048400 con 0x7f4d1809e8e0 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 client.28509.objecter ms_dispatch 0x7f4d18000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 3 client.28509.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 3 client.28509.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 20 client.28509.objecter dump_active .. 0 homeless 2024-09-16T10:53:55.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 client.28509.objecter ms_handle_connect 0x7f4d181c7000 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 client.28509.objecter resend_mon_ops 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d17fff700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 0x7f4d181c73e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009940 tx=0x7f4d08004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 1 --2- 172.21.15.5:0/2791729314 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4cf8036f60 0x7f4cf8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d0ffff700 10 monclient: get_auth_request con 0x7f4cf8036f60 auth_method 0 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d08020070 con 0x7f4d181c7000 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.782+0000 7f4d157fa700 10 monclient: dump: 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.808 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 10 client.28509.objecter ms_handle_connect 0x7f4cf8036f60 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d0ffff700 1 --2- 172.21.15.5:0/2791729314 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4cf8036f60 0x7f4cf8039420 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a3280 tx=0x7f4d00007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f4d0800be50 con 0x7f4d181c7000 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 msgr2=0x7f4d181c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 0x7f4d181c73e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009940 tx=0x7f4d08004770 comp rx=0 tx=0).stop 2024-09-16T10:53:55.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 msgr2=0x7f4d181c73e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.786+0000 7f4d157fa700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7000 0x7f4d181c73e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d1d5ae700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 0x7f4d1806dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d1d5ae700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1809f590 con 0x7f4d181c7920 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d17fff700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 0x7f4d1806dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d17fff700 10 monclient: get_auth_request con 0x7f4d181c7920 auth_method 0 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.794+0000 7f4d17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:55.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 10 monclient(hunting): handle_auth_done global_id 44146 payload 824 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 1 -- 172.21.15.5:0/2791729314 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d18120f50 con 0x7f4d181c7920 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 client.28509.objecter ms_handle_connect 0x7f4d181c7920 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 client.28509.objecter resend_mon_ops 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d17fff700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 0x7f4d1806dc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000c00 tx=0x7f4d08011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0801a440 con 0x7f4d181c7920 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 monclient: dump: 2024-09-16T10:53:55.813 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f4d0801a5a0 con 0x7f4d181c7920 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:55.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 msgr2=0x7f4d1806dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 0x7f4d1806dc60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000c00 tx=0x7f4d08011000 comp rx=0 tx=0).stop 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 msgr2=0x7f4d1806dc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d157fa700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d181c7920 0x7f4d1806dc60 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000c00 tx=0x7f4d08011000 comp rx=0 tx=0).stop 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 24, 2024-09-16T10:53:55.815 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 5, 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.816 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.817 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:53:55.818 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:53:55.819 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.820 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.822 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:53:55.823 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:53:55.824 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.826 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:53:55.827 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:53:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 10 librados: watch_flush enter 2024-09-16T10:53:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 10 librados: watch_flush exit 2024-09-16T10:53:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 20 client.28509.objecter shutdown clearing up homeless session... 2024-09-16T10:53:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 10 client.28509.objecter successfully canceled tick 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4cf8036f60 msgr2=0x7f4cf8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/2791729314 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4cf8036f60 0x7f4cf8039420 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 crypto rx=0x7f4d180a3280 tx=0x7f4d00007ee0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 10 monclient: shutdown 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 msgr2=0x7f4d18133240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 secure :-1 s=READY pgs=6759 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000cbc0 tx=0x7f4d1000ced0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1cdad700 1 -- 172.21.15.5:0/2791729314 reap_dead start 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 shutdown_connections 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 --2- 172.21.15.5:0/2791729314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d1809e8e0 0x7f4d18133240 unknown :-1 s=CLOSED pgs=6759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 >> 172.21.15.5:0/2791729314 conn(0x7f4d180a45d0 msgr2=0x7f4d18133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 shutdown_connections 2024-09-16T10:53:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 -- 172.21.15.5:0/2791729314 wait complete. 2024-09-16T10:53:55.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:55.798+0000 7f4d1d5ae700 1 librados: shutdown 2024-09-16T10:53:55.831 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:53:55.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T10:53:55.831 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:53:55.831 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T10:53:55.832 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T10:53:55.832 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T10:53:55.832 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:53:55.902 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T10:53:55.902 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:53:55.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:53:56.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: get_monmap_and_config 2024-09-16T10:53:56.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: build_initial_monmap 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: monmap: 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:56.017387+0000 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:56.017387+0000 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:56.018 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 Processor -- start 2024-09-16T10:53:56.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 -- start start 2024-09-16T10:53:56.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: init 2024-09-16T10:53:56.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:56.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f554409aa50 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5544071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55441386f0 0x7f554413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-c con 0x7f55441386f0 addr v2:172.21.15.143:3300/0 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554409caa0 con 0x7f554409aa50 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554409ae30 con 0x7f5544071870 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554409e3e0 con 0x7f55441386f0 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 10 monclient(hunting): authenticate will time out at 1094.594003s 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f55437fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f554409aa50 0x7f554409e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59200/0 (socket says 172.21.15.5:59200) 2024-09-16T10:53:56.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f55437fe700 1 -- 172.21.15.5:0/401461504 learned_addr learned my addr 172.21.15.5:0/401461504 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 1 --2- 172.21.15.5:0/401461504 >> v2:172.21.15.143:3300/0 conn(0x7f55441386f0 0x7f554413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): get_auth_request con 0x7f55441386f0 auth_method 0 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242155966 0 0) 0x7f5534001110 con 0x7f554409aa50 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): _negotiate 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:56.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): my global_id is 28512 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55300035d0 con 0x7f554409aa50 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3062407596 0 0) 0x7f55300035d0 con 0x7f554409aa50 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 >> v2:172.21.15.143:3300/0 conn(0x7f55441386f0 msgr2=0x7f554413aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 --2- 172.21.15.5:0/401461504 >> v2:172.21.15.143:3300/0 conn(0x7f55441386f0 0x7f554413aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 >> v1:172.21.15.143:6789/0 conn(0x7f5544071870 legacy=0x7f554409eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554409f700 con 0x7f554409aa50 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _finish_auth 0 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f5534001800 con 0x7f554409aa50 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 5 monclient: authenticate success, global_id 28512 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:56.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: dump: 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 >> v1:172.21.15.5:6789/0 conn(0x7f554409aa50 legacy=0x7f554409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530003a20 0x7f5530003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): picked mon.c con 0x7f5530003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5530004870 0x7f5530004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): picked mon.b con 0x7f5530004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 0x7f553000cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): picked mon.a con 0x7f5530005350 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554409e3e0 con 0x7f5530005350 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553000d330 con 0x7f5530004870 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553000d4f0 con 0x7f5530003a20 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5542ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5548ffc700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f55437fe700 1 -- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530003a20 msgr2=0x7f5530003e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f55437fe700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530003a20 0x7f5530003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f553bfff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5530004870 0x7f5530004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 0x7f553000cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f553bfff700 10 monclient(hunting): get_auth_request con 0x7f5530004870 auth_method 0 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f553bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f553bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f553bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): get_auth_request con 0x7f5530005350 auth_method 0 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.014+0000 7f5543fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:56.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient(hunting): handle_auth_done global_id 28512 payload 293 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 -- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530003a20 msgr2=0x7f5530003e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530003a20 0x7f5530003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 -- 172.21.15.5:0/401461504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5530004870 msgr2=0x7f5530004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5530004870 0x7f5530004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 monclient: found mon.a 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 -- 172.21.15.5:0/401461504 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55300035d0 con 0x7f5530005350 2024-09-16T10:53:56.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient: _finish_auth 0 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 10 monclient: _check_auth_tickets 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f553bfff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5530004870 0x7f5530004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 -- 172.21.15.5:0/401461504 reap_dead start 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5543fff700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 0x7f553000cb40 secure :-1 s=READY pgs=6768 cs=0 l=1 rev1=1 crypto rx=0x7f553c009fc0 tx=0x7f55300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f489a4511ef3ede server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c003a40 con 0x7f5530005350 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: dump: 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c00f040 con 0x7f5530005350 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 1 -- 172.21.15.5:0/401461504 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c003dc0 con 0x7f5530005350 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 4 set_mon_vals no callback set 2024-09-16T10:53:56.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5542ffd700 10 monclient: dump: 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:56.035 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 10 monclient: shutdown 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 -- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 msgr2=0x7f553000cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 0x7f553000cb40 secure :-1 s=READY pgs=6768 cs=0 l=1 rev1=1 crypto rx=0x7f553c009fc0 tx=0x7f55300058a0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 -- 172.21.15.5:0/401461504 shutdown_connections 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 --2- 172.21.15.5:0/401461504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5530005350 0x7f553000cb40 unknown :-1 s=CLOSED pgs=6768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 -- 172.21.15.5:0/401461504 >> 172.21.15.5:0/401461504 conn(0x7f55440a45d0 msgr2=0x7f55440a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 -- 172.21.15.5:0/401461504 shutdown_connections 2024-09-16T10:53:56.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 -- 172.21.15.5:0/401461504 wait complete. 2024-09-16T10:53:56.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 10 monclient: build_initial_monmap 2024-09-16T10:53:56.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 10 monclient: monmap: 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 librados: starting msgr at 2024-09-16T10:53:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 librados: starting objecter 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.018+0000 7f5548ffc700 1 Processor -- start 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- start start 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 librados: setting wanted keys 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 librados: calling monclient init 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: init 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 0x7f554411a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-c con 0x7f5530005350 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-a con 0x7f554411aa70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55441cc8b0 0x7f55441ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): picked mon.noname-b con 0x7f55441cc8b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554411b390 con 0x7f554411aa70 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f55437fe700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 msgr2=0x7f554411a530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55440a3260 con 0x7f55441cc8b0 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554411daf0 con 0x7f5530005350 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f55437fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 0x7f554411a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient(hunting): authenticate will time out at 1094.600750s 2024-09-16T10:53:56.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60956/0 (socket says 172.21.15.5:60956) 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 -- 172.21.15.5:0/549655823 learned_addr learned my addr 172.21.15.5:0/549655823 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): get_auth_request con 0x7f554411aa70 auth_method 0 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5543fff700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55441cc8b0 0x7f55441ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5543fff700 10 monclient(hunting): get_auth_request con 0x7f55441cc8b0 auth_method 0 2024-09-16T10:53:56.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5543fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5543fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5543fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient(hunting): handle_auth_done global_id 28515 payload 824 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 -- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 msgr2=0x7f554411a530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 0x7f554411a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 -- 172.21.15.5:0/549655823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55441cc8b0 msgr2=0x7f55441ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55441cc8b0 0x7f55441ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 monclient: found mon.noname-a 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 -- 172.21.15.5:0/549655823 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554409b0d0 con 0x7f554411aa70 2024-09-16T10:53:56.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f553bfff700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 secure :-1 s=READY pgs=6769 cs=0 l=1 rev1=1 crypto rx=0x7f552c00f770 tx=0x7f552c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 client.?.objecter ms_handle_connect 0x7f554411aa70 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0052e0 con 0x7f554411aa70 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: dump: 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.046 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: _finish_auth 0 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 5 monclient: authenticate success, global_id 28515 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c005440 con 0x7f554411aa70 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0038e0 con 0x7f554411aa70 2024-09-16T10:53:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _renew_subs 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554411b720 con 0x7f554411aa70 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5541ffb700 4 set_mon_vals no callback set 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 monclient: dump: 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 client.28515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _renew_subs 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554411c510 con 0x7f554411aa70 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 librados: init done 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554411d8a0 con 0x7f554411aa70 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f552c003a40 con 0x7f554411aa70 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 --2- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 0x7f55240393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f552c048310 con 0x7f554411aa70 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 10 client.28515.objecter ms_dispatch 0x7f5544000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 3 client.28515.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 3 client.28515.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f5540ff9700 20 client.28515.objecter dump_active .. 0 homeless 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f55437fe700 1 --2- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 0x7f55240393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.022+0000 7f55437fe700 10 monclient: get_auth_request con 0x7f5524036f10 auth_method 0 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.026+0000 7f5540ff9700 10 client.28515.objecter ms_handle_connect 0x7f5524036f10 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.026+0000 7f55437fe700 1 --2- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 0x7f55240393d0 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f5534001b90 tx=0x7f5534001c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.034+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f552c012760 con 0x7f554411aa70 2024-09-16T10:53:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.034+0000 7f5540ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.034+0000 7f5540ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5548ffc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:53:56.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5548ffc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5548ffc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f554411d680 con 0x7f554411aa70 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5540ff9700 1 -- 172.21.15.5:0/549655823 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f552c01c3c0 con 0x7f554411aa70 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5540ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.270+0000 7f5540ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:56.275 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 10 librados: watch_flush enter 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 10 librados: watch_flush exit 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 20 client.28515.objecter shutdown clearing up homeless session... 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 10 client.28515.objecter successfully canceled tick 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 msgr2=0x7f55240393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 0x7f55240393d0 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f5534001b90 tx=0x7f5534001c40 comp rx=0 tx=0).stop 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 10 monclient: shutdown 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 msgr2=0x7f554411ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 secure :-1 s=READY pgs=6769 cs=0 l=1 rev1=1 crypto rx=0x7f552c00f770 tx=0x7f552c00fb30 comp rx=0 tx=0).stop 2024-09-16T10:53:56.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 shutdown_connections 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55441cc8b0 0x7f55441ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5524036f10 0x7f55240393d0 unknown :-1 s=CLOSED pgs=3745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f554411aa70 0x7f554411ae50 unknown :-1 s=CLOSED pgs=6769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 --2- 172.21.15.5:0/549655823 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5530005350 0x7f554411a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 >> 172.21.15.5:0/549655823 conn(0x7f55440a45d0 msgr2=0x7f55441343f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 shutdown_connections 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 -- 172.21.15.5:0/549655823 wait complete. 2024-09-16T10:53:56.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:56.274+0000 7f5548ffc700 1 librados: shutdown 2024-09-16T10:53:59.295 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: get_monmap_and_config 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: build_initial_monmap 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: monmap: 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:59.415 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:53:59.414833+0000 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:53:59.414833+0000 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 Processor -- start 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 -- start start 2024-09-16T10:53:59.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: init 2024-09-16T10:53:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:59.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25d80a2bf0 addr v1:172.21.15.5:6789/0 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25d809f780 addr v1:172.21.15.143:6789/0 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f25d80a36e0 0x7f25d813e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-a con 0x7f25d80a36e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80a3c40 con 0x7f25d80a2bf0 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d80a4300 con 0x7f25d809f780 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d809fda0 con 0x7f25d80a36e0 2024-09-16T10:53:59.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dedcc700 10 monclient(hunting): authenticate will time out at 1097.991242s 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f25d80a36e0 0x7f25d813e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dddca700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f25d80a2bf0 0x7f25d80a2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59214/0 (socket says 172.21.15.5:59214) 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f25d80a36e0 0x7f25d813e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60960/0 (socket says 172.21.15.5:60960) 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25dddca700 1 -- 172.21.15.5:0/3987407496 learned_addr learned my addr 172.21.15.5:0/3987407496 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): get_auth_request con 0x7f25d80a36e0 auth_method 0 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:59.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.409+0000 7f25de5cb700 10 monclient(hunting): handle_auth_done global_id 28518 payload 274 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 1 -- 172.21.15.5:0/3987407496 >> v1:172.21.15.143:6789/0 conn(0x7f25d809f780 legacy=0x7f25d813ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 1 -- 172.21.15.5:0/3987407496 >> v1:172.21.15.5:6789/0 conn(0x7f25d80a2bf0 legacy=0x7f25d80a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 10 monclient: _finish_hunting 0 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 1 monclient: found mon.noname-a 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 1 -- 172.21.15.5:0/3987407496 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d80a0080 con 0x7f25d80a36e0 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25de5cb700 1 --2- 172.21.15.5:0/3987407496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813e530 secure :-1 s=READY pgs=6775 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009d90 tx=0x7f25d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f58be89592b172 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 1 -- 172.21.15.5:0/3987407496 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d000e070 con 0x7f25d80a36e0 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:53:59.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: dump: 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:59.424 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: _finish_auth 0 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: _check_auth_tickets 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 1 -- 172.21.15.5:0/3987407496 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d000bbd0 con 0x7f25d80a36e0 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 1 -- 172.21.15.5:0/3987407496 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0012430 con 0x7f25d80a36e0 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dcdc8700 10 monclient: dump: 2024-09-16T10:53:59.425 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 5 monclient: authenticate success, global_id 28518 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 10 monclient: get_monmap_and_config success 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 4 set_mon_vals no callback set 2024-09-16T10:53:59.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 10 monclient: shutdown 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 -- 172.21.15.5:0/3987407496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 msgr2=0x7f25d813e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 --2- 172.21.15.5:0/3987407496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813e530 secure :-1 s=READY pgs=6775 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009d90 tx=0x7f25d000b330 comp rx=0 tx=0).stop 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 -- 172.21.15.5:0/3987407496 shutdown_connections 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 --2- 172.21.15.5:0/3987407496 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813e530 unknown :-1 s=CLOSED pgs=6775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 -- 172.21.15.5:0/3987407496 >> 172.21.15.5:0/3987407496 conn(0x7f25d80a53e0 msgr2=0x7f25d8073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 -- 172.21.15.5:0/3987407496 shutdown_connections 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 -- 172.21.15.5:0/3987407496 wait complete. 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 10 monclient: build_initial_monmap 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 10 monclient: monmap: 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:53:59.427 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 librados: starting msgr at 2024-09-16T10:53:59.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.413+0000 7f25dedcc700 1 librados: starting objecter 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 Processor -- start 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- start start 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 librados: setting wanted keys 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 librados: calling monclient init 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: init 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _reopen_session rank -1 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 0x7f25d8135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:59.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-b con 0x7f25d809f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 0x7f25d8135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-c con 0x7f25d80a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): picked mon.noname-a con 0x7f25d80a36e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8136130 con 0x7f25d80a36e0 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d813c650 con 0x7f25d809f780 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d811f050 con 0x7f25d80a2bf0 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): _renew_subs 2024-09-16T10:53:59.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dd5c9700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 msgr2=0x7f25d8135860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient(hunting): authenticate will time out at 1097.997503s 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dd5c9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 0x7f25d8135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 0x7f25d8135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60976/0 (socket says 172.21.15.5:60976) 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 1 -- 172.21.15.5:0/1392182531 learned_addr learned my addr 172.21.15.5:0/1392182531 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): get_auth_request con 0x7f25d809f780 auth_method 0 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): get_auth_request con 0x7f25d80a36e0 auth_method 0 2024-09-16T10:53:59.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient(hunting): handle_auth_done global_id 44155 payload 824 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 -- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 msgr2=0x7f25d8135860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 0x7f25d8135860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 -- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 msgr2=0x7f25d813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient: _finish_hunting 0 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 monclient: found mon.noname-b 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 -- 172.21.15.5:0/1392182531 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0009a70 con 0x7f25d809f780 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25de5cb700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 client.?.objecter ms_handle_connect 0x7f25d809f780 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dddca700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 0x7f25d8135320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25c800c4b0 tx=0x7f25c800c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8005d90 con 0x7f25d809f780 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:59.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: dump: 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:59.438 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: _finish_auth 0 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c80070d0 con 0x7f25d809f780 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 5 monclient: authenticate success, global_id 44155 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _renew_subs 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d811f330 con 0x7f25d809f780 2024-09-16T10:53:59.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8007440 con 0x7f25d809f780 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25ce7fc700 10 monclient: dump: 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:53:59.440 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25cffff700 4 set_mon_vals no callback set 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 client.44155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _renew_subs 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d8135da0 con 0x7f25d809f780 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.417+0000 7f25dedcc700 1 librados: init done 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dedcc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dedcc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dedcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d8135f30 con 0x7f25d809f780 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f25c80075a0 con 0x7f25d809f780 2024-09-16T10:53:59.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 1 --2- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 0x7f25bc0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f25c8047740 con 0x7f25d809f780 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 10 client.44155.objecter ms_dispatch 0x7f25d8000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 3 client.44155.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 3 client.44155.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dd5c9700 1 --2- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 0x7f25bc0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 20 client.44155.objecter dump_active .. 0 homeless 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dd5c9700 10 monclient: get_auth_request con 0x7f25bc036f10 auth_method 0 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25ce7fc700 10 client.44155.objecter ms_handle_connect 0x7f25bc036f10 2024-09-16T10:53:59.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.421+0000 7f25dd5c9700 1 --2- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 0x7f25bc0393d0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f25d806bae0 tx=0x7f25d4007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:53:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.433+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f25c8019b40 con 0x7f25d809f780 2024-09-16T10:53:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.433+0000 7f25ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:53:59.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.433+0000 7f25ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:53:59.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25dedcc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:53:59.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25dedcc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25dedcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:53:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f25d80a2fd0 con 0x7f25d809f780 2024-09-16T10:53:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25ce7fc700 1 -- 172.21.15.5:0/1392182531 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f25c8017020 con 0x7f25d809f780 2024-09-16T10:53:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:53:59.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.669+0000 7f25ce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:53:59.676 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 10 librados: watch_flush enter 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 10 librados: watch_flush exit 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 20 client.44155.objecter shutdown clearing up homeless session... 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 10 client.44155.objecter successfully canceled tick 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 msgr2=0x7f25bc0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 0x7f25bc0393d0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f25d806bae0 tx=0x7f25d4007a80 comp rx=0 tx=0).stop 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 10 monclient: shutdown 2024-09-16T10:53:59.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 msgr2=0x7f25d8135320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 0x7f25d8135320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25c800c4b0 tx=0x7f25c800c7c0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 shutdown_connections 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> v2:172.21.15.143:6812/1098205034 conn(0x7f25bc036f10 0x7f25bc0393d0 unknown :-1 s=CLOSED pgs=3746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25d80a36e0 0x7f25d813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f25d80a2bf0 0x7f25d8135860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 --2- 172.21.15.5:0/1392182531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25d809f780 0x7f25d8135320 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 >> 172.21.15.5:0/1392182531 conn(0x7f25d80a53e0 msgr2=0x7f25d806ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 shutdown_connections 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 -- 172.21.15.5:0/1392182531 wait complete. 2024-09-16T10:53:59.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:53:59.673+0000 7f25dedcc700 1 librados: shutdown 2024-09-16T10:54:02.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:54:02.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.805+0000 7f2825636700 10 monclient: get_monmap_and_config 2024-09-16T10:54:02.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.805+0000 7f2825636700 10 monclient: build_initial_monmap 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.805+0000 7f2825636700 10 monclient: monmap: 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:54:02.812642+0000 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:54:02.812642+0000 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:54:02.813 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:54:02.814 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 Processor -- start 2024-09-16T10:54:02.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 -- start start 2024-09-16T10:54:02.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient: init 2024-09-16T10:54:02.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28200a19f0 addr v1:172.21.15.143:6789/0 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 0x7f282013abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): picked mon.noname-b con 0x7f28200a24e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f282013b100 0x7f282013d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): picked mon.noname-c con 0x7f282013b100 addr v2:172.21.15.143:3300/0 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2817fff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 msgr2=0x7f282013abc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2817fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 0x7f282013abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282009e0b0 con 0x7f28200a19f0 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282009e2f0 con 0x7f282013b100 2024-09-16T10:54:02.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282009f9f0 con 0x7f28200a24e0 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2825636700 10 monclient(hunting): authenticate will time out at 1101.390848s 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f281ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f28200a19f0 0x7f28200a1dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:41776/0 (socket says 172.21.15.5:41776) 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f281ffff700 1 -- 172.21.15.5:0/3409209477 learned_addr learned my addr 172.21.15.5:0/3409209477 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 --2- 172.21.15.5:0/3409209477 >> v2:172.21.15.143:3300/0 conn(0x7f282013b100 0x7f282013d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): get_auth_request con 0x7f282013b100 auth_method 0 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:02.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient(hunting): handle_auth_done global_id 44158 payload 274 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 -- 172.21.15.5:0/3409209477 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 msgr2=0x7f282013abc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 --2- 172.21.15.5:0/3409209477 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 0x7f282013abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 -- 172.21.15.5:0/3409209477 >> v1:172.21.15.143:6789/0 conn(0x7f28200a19f0 legacy=0x7f28200a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient: _finish_hunting 0 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 monclient: found mon.noname-c 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.809+0000 7f2824e35700 1 -- 172.21.15.5:0/3409209477 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282009f150 con 0x7f282013b100 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2824e35700 1 --2- 172.21.15.5:0/3409209477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f282013b100 0x7f282013d4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673cb091e5e77258 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:02.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 1 -- 172.21.15.5:0/3409209477 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281800e070 con 0x7f282013b100 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: dump: 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:02.821 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: _finish_auth 0 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 5 monclient: authenticate success, global_id 44158 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 1 -- 172.21.15.5:0/3409209477 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281800bbd0 con 0x7f282013b100 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: get_monmap_and_config success 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 4 set_mon_vals no callback set 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 1 -- 172.21.15.5:0/3409209477 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818012430 con 0x7f282013b100 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:02.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281f7fe700 10 monclient: dump: 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: shutdown 2024-09-16T10:54:02.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- 172.21.15.5:0/3409209477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f282013b100 msgr2=0x7f282013d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- 172.21.15.5:0/3409209477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f282013b100 0x7f282013d4f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281800b330 comp rx=0 tx=0).stop 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- 172.21.15.5:0/3409209477 shutdown_connections 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- 172.21.15.5:0/3409209477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f282013b100 0x7f282013d4f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- 172.21.15.5:0/3409209477 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f28200a24e0 0x7f282013abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- 172.21.15.5:0/3409209477 >> 172.21.15.5:0/3409209477 conn(0x7f28200a3200 msgr2=0x7f2820073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- 172.21.15.5:0/3409209477 shutdown_connections 2024-09-16T10:54:02.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- 172.21.15.5:0/3409209477 wait complete. 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: build_initial_monmap 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: monmap: 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:54:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 librados: starting msgr at 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 librados: starting objecter 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 Processor -- start 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- start start 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 librados: setting wanted keys 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 librados: calling monclient init 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: init 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:54:02.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): picked mon.noname-a con 0x7f28200a19f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28200a24e0 0x7f2820120a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): picked mon.noname-b con 0x7f28200a24e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 0x7f2820120f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): picked mon.noname-c con 0x7f282013b100 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2820122970 con 0x7f28200a19f0 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2820122d10 con 0x7f28200a24e0 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2820122f90 con 0x7f282013b100 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:02.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2825636700 10 monclient(hunting): authenticate will time out at 1101.394953s 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2824e35700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 msgr2=0x7f2820120f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2824e35700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 0x7f2820120f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47484/0 (socket says 172.21.15.5:47484) 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 -- 172.21.15.5:0/2216481771 learned_addr learned my addr 172.21.15.5:0/2216481771 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28200a24e0 0x7f2820120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): get_auth_request con 0x7f28200a19f0 auth_method 0 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): get_auth_request con 0x7f28200a24e0 auth_method 0 2024-09-16T10:54:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f2817fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient(hunting): handle_auth_done global_id 28527 payload 824 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 -- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 msgr2=0x7f2820120f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 0x7f2820120f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 -- 172.21.15.5:0/2216481771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28200a24e0 msgr2=0x7f2820120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:02.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28200a24e0 0x7f2820120a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 monclient: found mon.noname-a 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.813+0000 7f281ffff700 1 -- 172.21.15.5:0/2216481771 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818009a70 con 0x7f28200a19f0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 client.?.objecter ms_handle_connect 0x7f28200a19f0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281ffff700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 crypto rx=0x7f281000c700 tx=0x7f281000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810005d90 con 0x7f28200a19f0 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:54:02.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: dump: 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: _finish_auth 0 2024-09-16T10:54:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: _check_auth_tickets 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 5 monclient: authenticate success, global_id 28527 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28100070d0 con 0x7f28200a19f0 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810007440 con 0x7f28200a19f0 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _renew_subs 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281effd700 4 set_mon_vals no callback set 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28201223c0 con 0x7f28200a19f0 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 monclient: dump: 2024-09-16T10:54:02.838 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 client.28527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _renew_subs 2024-09-16T10:54:02.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2820124160 con 0x7f28200a19f0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 1 librados: init done 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2820074670 con 0x7f28200a19f0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f28100075a0 con 0x7f28200a19f0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 --2- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 0x7f28000393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f28100477c0 con 0x7f28200a19f0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 client.28527.objecter ms_dispatch 0x7f2820000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 3 client.28527.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:54:02.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 3 client.28527.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 20 client.28527.objecter dump_active .. 0 homeless 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2817fff700 1 --2- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 0x7f28000393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2817fff700 10 monclient: get_auth_request con 0x7f2800036f10 auth_method 0 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f281d7fa700 10 client.28527.objecter ms_handle_connect 0x7f2800036f10 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.817+0000 7f2817fff700 1 --2- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 0x7f28000393d0 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f2808009470 tx=0x7f2808009400 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.825+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2810011d70 con 0x7f28200a19f0 2024-09-16T10:54:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.825+0000 7f281d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:02.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:02.825+0000 7f281d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:54:03.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f2825636700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:54:03.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f2825636700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:54:03.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f2825636700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:03.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28200721d0 con 0x7f28200a19f0 2024-09-16T10:54:03.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f281d7fa700 1 -- 172.21.15.5:0/2216481771 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7f2810017020 con 0x7f28200a19f0 2024-09-16T10:54:03.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f281d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:54:03.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.057+0000 7f281d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:54:03.063 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:54:03.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 10 librados: watch_flush enter 2024-09-16T10:54:03.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 10 librados: watch_flush exit 2024-09-16T10:54:03.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 20 client.28527.objecter shutdown clearing up homeless session... 2024-09-16T10:54:03.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 10 client.28527.objecter successfully canceled tick 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 msgr2=0x7f28000393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 0x7f28000393d0 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f2808009470 tx=0x7f2808009400 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 10 monclient: shutdown 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 msgr2=0x7f2820137fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 crypto rx=0x7f281000c700 tx=0x7f281000ca10 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 shutdown_connections 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f282013b100 0x7f2820120f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28200a24e0 0x7f2820120a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2800036f10 0x7f28000393d0 unknown :-1 s=CLOSED pgs=3747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 --2- 172.21.15.5:0/2216481771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f28200a19f0 0x7f2820137fe0 unknown :-1 s=CLOSED pgs=6777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:03.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 >> 172.21.15.5:0/2216481771 conn(0x7f28200a3200 msgr2=0x7f282009a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:03.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 shutdown_connections 2024-09-16T10:54:03.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 -- 172.21.15.5:0/2216481771 wait complete. 2024-09-16T10:54:03.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:03.061+0000 7f2825636700 1 librados: shutdown 2024-09-16T10:54:06.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:54:06.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.189+0000 7fdb1359e700 10 monclient: get_monmap_and_config 2024-09-16T10:54:06.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.189+0000 7fdb1359e700 10 monclient: build_initial_monmap 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.189+0000 7fdb1359e700 10 monclient: monmap: 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:54:06.196486+0000 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:54:06.196486+0000 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:54:06.197 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:06.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 Processor -- start 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 -- start start 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient: init 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb1409e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdb1409d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdb1409dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb14070e40 con 0x7fdb1409e8e0 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb140a01d0 con 0x7fdb1409dfd0 2024-09-16T10:54:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1409fef0 con 0x7fdb1409d6a0 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1359e700 10 monclient(hunting): authenticate will time out at 1104.773051s 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb09d9b700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 msgr2=0x7fdb14138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb09d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1259c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdb1409e8e0 0x7fdb1409ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37224/0 (socket says 172.21.15.5:37224) 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 learned_addr learned my addr 172.21.15.5:0/3042535313 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request con 0x7fdb1409dfd0 auth_method 0 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:06.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:54:06.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:06.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:06.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.193+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:06.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 msgr2=0x7fdb14138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request con 0x7fdb1409dfd0 auth_method 0 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:06.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.397+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:54:06.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 msgr2=0x7fdb14138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request con 0x7fdb1409dfd0 auth_method 0 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:06.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:06.801+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 msgr2=0x7fdb14138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request con 0x7fdb1409dfd0 auth_method 0 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:07.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:07.605+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:54:09.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): tick 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 monclient(hunting): continuing hunt 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 msgr2=0x7fdb14138b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb1409d6a0 0x7fdb14138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 msgr2=0x7fdb1413b0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.5:3300/0 conn(0x7fdb1409dfd0 0x7fdb1413b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 -- 172.21.15.5:0/3042535313 >> v1:172.21.15.5:6789/0 conn(0x7fdb1409e8e0 legacy=0x7fdb1409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient: _add_conns ranks=[4,1] 2024-09-16T10:54:09.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): picked mon.noname-c con 0x7fdaf4003740 addr v2:172.21.15.143:3300/0 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaf40062b0 addr v1:172.21.15.143:6789/0 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 -- 172.21.15.5:0/3042535313 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb1409f9f0 con 0x7fdaf40062b0 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 1 -- 172.21.15.5:0/3042535313 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb140a01d0 con 0x7fdaf4003740 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb10d99700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 10 monclient(hunting): get_auth_request con 0x7fdaf4003740 auth_method 0 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:54:09.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:09.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.193+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 10 monclient(hunting): get_auth_request con 0x7fdaf4003740 auth_method 0 2024-09-16T10:54:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:09.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:54:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:09.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.397+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:54:09.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 10 monclient(hunting): get_auth_request con 0x7fdaf4003740 auth_method 0 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:09.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:09.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:54:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 10 monclient(hunting): get_auth_request con 0x7fdaf4003740 auth_method 0 2024-09-16T10:54:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:54:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.605+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047187762 0 0) 0x7fdb1409f9f0 con 0x7fdaf40062b0 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 10 monclient(hunting): _negotiate 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 10 monclient(hunting): my global_id is 44164 2024-09-16T10:54:10.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.777+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb000035d0 con 0x7fdaf40062b0 2024-09-16T10:54:10.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469977773 0 0) 0x7fdb000035d0 con 0x7fdaf40062b0 2024-09-16T10:54:10.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 msgr2=0x7fdaf4005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 --2- 172.21.15.5:0/3042535313 >> v2:172.21.15.143:3300/0 conn(0x7fdaf4003740 0x7fdaf4005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 monclient: found mon.noname-c-legacy 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf400b060 con 0x7fdaf40062b0 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _finish_auth 0 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7fdafc0043f0 con 0x7fdaf40062b0 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 5 monclient: authenticate success, global_id 44164 2024-09-16T10:54:10.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: dump: 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:10.803 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 >> v1:172.21.15.143:6789/0 conn(0x7fdaf40062b0 legacy=0x7fdaf4008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 1 -- 172.21.15.5:0/3042535313 reap_dead start 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 0x7fdb00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): picked mon.c con 0x7fdb00003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 0x7fdb00004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): picked mon.a con 0x7fdb00004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 0x7fdb0000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): picked mon.b con 0x7fdb00005280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:54:10.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140a01d0 con 0x7fdb00004870 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0000c1b0 con 0x7fdb00005280 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0000c370 con 0x7fdb00003a20 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1259c700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 msgr2=0x7fdb00003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1259c700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 0x7fdb00003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 0x7fdb00004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 0x7fdb0000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): get_auth_request con 0x7fdb00004870 auth_method 0 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request con 0x7fdb00005280 auth_method 0 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_done global_id 44164 payload 293 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 msgr2=0x7fdb00003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 0x7fdb00003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 msgr2=0x7fdb0000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 0x7fdb0000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 monclient: found mon.a 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 -- 172.21.15.5:0/3042535313 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000035d0 con 0x7fdb00004870 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient: _finish_auth 0 2024-09-16T10:54:10.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb12d9d700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 0x7fdb0000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 0x7fdb00004cf0 secure :-1 s=READY pgs=6780 cs=0 l=1 rev1=1 crypto rx=0x7fdafc003a30 tx=0x7fdb000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfa5f13e90870d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0060d0 con 0x7fdb00004870 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: dump: 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:10.808 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc006230 con 0x7fdb00004870 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 10 monclient: get_monmap_and_config success 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 1 -- 172.21.15.5:0/3042535313 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0045e0 con 0x7fdb00004870 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb1359e700 4 set_mon_vals no callback set 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:10.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.793+0000 7fdb11d9b700 10 monclient: dump: 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 10 monclient: shutdown 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:10.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 msgr2=0x7fdb00004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 0x7fdb00004cf0 secure :-1 s=READY pgs=6780 cs=0 l=1 rev1=1 crypto rx=0x7fdafc003a30 tx=0x7fdb000058a0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- 172.21.15.5:0/3042535313 shutdown_connections 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00005280 0x7fdb0000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00004870 0x7fdb00004cf0 unknown :-1 s=CLOSED pgs=6780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 --2- 172.21.15.5:0/3042535313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00003a20 0x7fdb00003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- 172.21.15.5:0/3042535313 >> 172.21.15.5:0/3042535313 conn(0x7fdb140a45d0 msgr2=0x7fdb140a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- 172.21.15.5:0/3042535313 shutdown_connections 2024-09-16T10:54:10.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- 172.21.15.5:0/3042535313 wait complete. 2024-09-16T10:54:10.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 10 monclient: build_initial_monmap 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 10 monclient: monmap: 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:54:10.816 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 librados: starting msgr at 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 librados: starting objecter 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 Processor -- start 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 -- start start 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 librados: setting wanted keys 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 1 librados: calling monclient init 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.797+0000 7fdb1359e700 10 monclient: init 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:10.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdb00003a20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00004870 0x7fdb141330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdb00004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 0x7fdb141c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): picked mon.noname-c con 0x7fdb00005280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1409b3e0 con 0x7fdb00003a20 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb141bdea0 con 0x7fdb00004870 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:10.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1409b0d0 con 0x7fdb00005280 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient(hunting): authenticate will time out at 1109.380852s 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb12d9d700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 msgr2=0x7fdb141c6bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb12d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 0x7fdb141c6bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47570/0 (socket says 172.21.15.5:47570) 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/4108303824 learned_addr learned my addr 172.21.15.5:0/4108303824 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): get_auth_request con 0x7fdb00003a20 auth_method 0 2024-09-16T10:54:10.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00004870 0x7fdb141330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): get_auth_request con 0x7fdb00004870 auth_method 0 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:10.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb09d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient(hunting): handle_auth_done global_id 28530 payload 824 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 msgr2=0x7fdb141c6bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 0x7fdb141c6bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/4108303824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00004870 msgr2=0x7fdb141330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00004870 0x7fdb141330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient: _finish_hunting 0 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 monclient: found mon.noname-a 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 -- 172.21.15.5:0/4108303824 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1409b8a0 con 0x7fdb00003a20 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1259c700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 secure :-1 s=READY pgs=6781 cs=0 l=1 rev1=1 crypto rx=0x7fdb04001290 tx=0x7fdb040015a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 client.?.objecter ms_handle_connect 0x7fdb00003a20 2024-09-16T10:54:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04017460 con 0x7fdb00003a20 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: dump: 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb040175c0 con 0x7fdb00003a20 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 5 monclient: authenticate success, global_id 28530 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _renew_subs 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1159a700 4 set_mon_vals no callback set 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb040179b0 con 0x7fdb00003a20 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1411f510 con 0x7fdb00003a20 2024-09-16T10:54:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb0b7fe700 10 monclient: dump: 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:10.824 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 client.28530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _renew_subs 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14120a80 con 0x7fdb00003a20 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.801+0000 7fdb1359e700 1 librados: init done 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb1359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb1359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb1359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb14075890 con 0x7fdb00003a20 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fdb04017b10 con 0x7fdb00003a20 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 1 --2- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 0x7fdaf4039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7fdb040491a0 con 0x7fdb00003a20 2024-09-16T10:54:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 10 client.28530.objecter ms_dispatch 0x7fdb14000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 3 client.28530.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 3 client.28530.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 20 client.28530.objecter dump_active .. 0 homeless 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 0x7fdaf4039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb09d9b700 10 monclient: get_auth_request con 0x7fdaf40377f0 auth_method 0 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb0b7fe700 10 client.28530.objecter ms_handle_connect 0x7fdaf40377f0 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.805+0000 7fdb09d9b700 1 --2- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 0x7fdaf4039cb0 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000c00 tx=0x7fdafc00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:10.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.813+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdb040159a0 con 0x7fdb00003a20 2024-09-16T10:54:10.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.813+0000 7fdb0b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:10.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:10.813+0000 7fdb0b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:54:11.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb1359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:54:11.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb1359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:54:11.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb1359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdb1409fef0 con 0x7fdb00003a20 2024-09-16T10:54:11.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb0b7fe700 1 -- 172.21.15.5:0/4108303824 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fdb0401b070 con 0x7fdb00003a20 2024-09-16T10:54:11.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb0b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:54:11.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.045+0000 7fdb0b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:54:11.053 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 10 librados: watch_flush enter 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 10 librados: watch_flush exit 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 20 client.28530.objecter shutdown clearing up homeless session... 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 10 client.28530.objecter successfully canceled tick 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 msgr2=0x7fdaf4039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 0x7fdaf4039cb0 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000c00 tx=0x7fdafc00a000 comp rx=0 tx=0).stop 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 10 monclient: shutdown 2024-09-16T10:54:11.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 msgr2=0x7fdb141ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 secure :-1 s=READY pgs=6781 cs=0 l=1 rev1=1 crypto rx=0x7fdb04001290 tx=0x7fdb040015a0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 shutdown_connections 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdb00005280 0x7fdb141c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb00004870 0x7fdb141330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdaf40377f0 0x7fdaf4039cb0 unknown :-1 s=CLOSED pgs=3748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 --2- 172.21.15.5:0/4108303824 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb00003a20 0x7fdb141ca0c0 unknown :-1 s=CLOSED pgs=6781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 >> 172.21.15.5:0/4108303824 conn(0x7fdb140a45d0 msgr2=0x7fdb14133cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 shutdown_connections 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 -- 172.21.15.5:0/4108303824 wait complete. 2024-09-16T10:54:11.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.049+0000 7fdb1359e700 1 librados: shutdown 2024-09-16T10:54:11.068 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T10:54:11.068 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T10:54:11.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:54:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 10 monclient: get_monmap_and_config 2024-09-16T10:54:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 10 monclient: build_initial_monmap 2024-09-16T10:54:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 10 monclient: monmap: 2024-09-16T10:54:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:54:11.183821+0000 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:54:11.183821+0000 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.185 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:54:11.186 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:54:11.186 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:54:11.186 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:54:11.186 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:54:11.186 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 1 Processor -- start 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 1 -- start start 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.177+0000 7f5757fff700 10 monclient: init 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57580a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f575809f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57580a36e0 0x7f575813e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-c con 0x7f57580a36e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580a3c40 con 0x7f575809f780 2024-09-16T10:54:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580a4300 con 0x7f57580a2bf0 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575809fda0 con 0x7f57580a36e0 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient(hunting): authenticate will time out at 1109.760417s 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57567fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f575809f780 0x7f575813ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37226/0 (socket says 172.21.15.5:37226) 2024-09-16T10:54:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57567fc700 1 -- 172.21.15.5:0/3778750273 learned_addr learned my addr 172.21.15.5:0/3778750273 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 --2- 172.21.15.5:0/3778750273 >> v2:172.21.15.143:3300/0 conn(0x7f57580a36e0 0x7f575813e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): get_auth_request con 0x7f57580a36e0 auth_method 0 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561775284 0 0) 0x7f57580a3c40 con 0x7f575809f780 2024-09-16T10:54:11.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient(hunting): _negotiate 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient(hunting): my global_id is 28533 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57440035d0 con 0x7f575809f780 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient(hunting): handle_auth_done global_id 44170 payload 274 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397501295 0 0) 0x7f57440035d0 con 0x7f575809f780 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 -- 172.21.15.5:0/3778750273 >> v1:172.21.15.143:6789/0 conn(0x7f57580a2bf0 legacy=0x7f57580a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 -- 172.21.15.5:0/3778750273 >> v1:172.21.15.5:6789/0 conn(0x7f575809f780 legacy=0x7f575813ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient: _finish_hunting 0 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 monclient: found mon.noname-c 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 -- 172.21.15.5:0/3778750273 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580a0080 con 0x7f57580a36e0 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f57577fe700 1 --2- 172.21.15.5:0/3778750273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813e530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5750009d90 tx=0x7f575000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6fb3d277500653 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575000e070 con 0x7f57580a36e0 2024-09-16T10:54:11.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: dump: 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: _finish_auth 0 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 5 monclient: authenticate success, global_id 44170 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5750003a40 con 0x7f57580a36e0 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 1 -- 172.21.15.5:0/3778750273 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740001670 con 0x7f57580a36e0 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 4 set_mon_vals no callback set 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5755ffb700 10 monclient: dump: 2024-09-16T10:54:11.210 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.211 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 10 monclient: shutdown 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 -- 172.21.15.5:0/3778750273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 msgr2=0x7f575813e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.181+0000 7f5757fff700 1 --2- 172.21.15.5:0/3778750273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813e530 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5750009d90 tx=0x7f575000b330 comp rx=0 tx=0).stop 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- 172.21.15.5:0/3778750273 shutdown_connections 2024-09-16T10:54:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 --2- 172.21.15.5:0/3778750273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813e530 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- 172.21.15.5:0/3778750273 >> 172.21.15.5:0/3778750273 conn(0x7f57580a53e0 msgr2=0x7f5758073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- 172.21.15.5:0/3778750273 shutdown_connections 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- 172.21.15.5:0/3778750273 wait complete. 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient: build_initial_monmap 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient: monmap: 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:11.213 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 librados: starting msgr at 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 librados: starting objecter 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 Processor -- start 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- start start 2024-09-16T10:54:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 librados: setting wanted keys 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 librados: calling monclient init 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient: init 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-a con 0x7f575809f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57580a2bf0 0x7f57581cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-c con 0x7f57580a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): picked mon.noname-b con 0x7f57580a36e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5758135d70 con 0x7f575809f780 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575811ef80 con 0x7f57580a36e0 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575813cf00 con 0x7f57580a2bf0 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:11.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5757fff700 10 monclient(hunting): authenticate will time out at 1109.767781s 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f57567fc700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57580a2bf0 msgr2=0x7f57581cb120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f57567fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57580a2bf0 0x7f57581cb120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47580/0 (socket says 172.21.15.5:47580) 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 1 -- 172.21.15.5:0/2605061356 learned_addr learned my addr 172.21.15.5:0/2605061356 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 10 monclient(hunting): get_auth_request con 0x7f575809f780 auth_method 0 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.185+0000 7f5756ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57577fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient(hunting): handle_auth_done global_id 28536 payload 824 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57580a2bf0 msgr2=0x7f57581cb120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f57580a2bf0 0x7f57581cb120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 msgr2=0x7f575813a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 monclient: found mon.noname-a 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:11.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750009a70 con 0x7f575809f780 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57577fe700 10 monclient: get_auth_request con 0x7f57580a36e0 auth_method 0 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57577fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57580a36e0 0x7f575813a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 client.?.objecter ms_handle_connect 0x7f575809f780 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5756ffd700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 secure :-1 s=READY pgs=6783 cs=0 l=1 rev1=1 crypto rx=0x7f574800c4e0 tx=0x7f574800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011790 con 0x7f575809f780 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:54:11.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: dump: 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: _finish_auth 0 2024-09-16T10:54:11.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 5 monclient: authenticate success, global_id 28536 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57480118f0 con 0x7f575809f780 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011ca0 con 0x7f575809f780 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 monclient: _renew_subs 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57557fa700 4 set_mon_vals no callback set 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575813df70 con 0x7f575809f780 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 client.28536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: dump: 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 monclient: _renew_subs 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575813cc80 con 0x7f575809f780 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 1 librados: init done 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 0x7f575813b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5758074d50 con 0x7f575813aca0 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 0x7f575813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient: get_auth_request con 0x7f575813aca0 auth_method 0 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5748005080 con 0x7f575809f780 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 10 monclient(hunting): handle_auth_done global_id 28539 payload 824 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 --2- 172.21.15.5:0/2605061356 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5738036f10 0x7f57380393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f575813ea70 con 0x7f575813aca0 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f5748047e90 con 0x7f575809f780 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 client.28536.objecter ms_dispatch 0x7f5758000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:54:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 3 client.28536.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 3 client.28536.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 20 client.28536.objecter dump_active .. 0 homeless 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 client.28536.objecter ms_handle_connect 0x7f575813aca0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 client.28536.objecter resend_mon_ops 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57567fc700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 0x7f575813b080 secure :-1 s=READY pgs=6784 cs=0 l=1 rev1=1 crypto rx=0x7f575813bf30 tx=0x7f574000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57577fe700 1 --2- 172.21.15.5:0/2605061356 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5738036f10 0x7f57380393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f57577fe700 10 monclient: get_auth_request con 0x7f5738036f10 auth_method 0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740004ca0 con 0x7f575813aca0 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.189+0000 7f574f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 10 monclient: dump: 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f5740004e00 con 0x7f575813aca0 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 msgr2=0x7f575813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 0x7f575813b080 secure :-1 s=READY pgs=6784 cs=0 l=1 rev1=1 crypto rx=0x7f575813bf30 tx=0x7f574000ca10 comp rx=0 tx=0).stop 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 msgr2=0x7f575813b080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813aca0 0x7f575813b080 unknown :-1 s=CLOSED pgs=6784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f574f7fe700 10 client.28536.objecter ms_handle_connect 0x7f5738036f10 2024-09-16T10:54:11.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.193+0000 7f57577fe700 1 --2- 172.21.15.5:0/2605061356 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5738036f10 0x7f57380393d0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f57580a7de0 tx=0x7f5750010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:54:11.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 0x7f5758136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575811e920 con 0x7f575813b5c0 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 0x7f5758136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient: get_auth_request con 0x7f575813b5c0 auth_method 0 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 10 monclient(hunting): handle_auth_done global_id 28542 payload 824 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 1 -- 172.21.15.5:0/2605061356 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f575809f510 con 0x7f575813b5c0 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57567fc700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 0x7f5758136de0 secure :-1 s=READY pgs=6785 cs=0 l=1 rev1=1 crypto rx=0x7f574000d010 tx=0x7f574000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 client.28536.objecter ms_handle_connect 0x7f575813b5c0 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 client.28536.objecter resend_mon_ops 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740020070 con 0x7f575813b5c0 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 monclient: dump: 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4768 (secure 0 0 0) 0x7f574001b110 con 0x7f575813b5c0 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 msgr2=0x7f5758136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 0x7f5758136de0 secure :-1 s=READY pgs=6785 cs=0 l=1 rev1=1 crypto rx=0x7f574000d010 tx=0x7f574000f000 comp rx=0 tx=0).stop 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 msgr2=0x7f5758136de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f574f7fe700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575813b5c0 0x7f5758136de0 unknown :-1 s=CLOSED pgs=6785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.232 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 28, 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:54:11.233 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:54:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.237 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.238 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.239 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.240 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.241 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:54:11.242 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.243 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.244 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 8 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.245 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:54:11.246 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:54:11.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 librados: watch_flush enter 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 librados: watch_flush exit 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 20 client.28536.objecter shutdown clearing up homeless session... 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 client.28536.objecter successfully canceled tick 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5738036f10 msgr2=0x7f57380393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 --2- 172.21.15.5:0/2605061356 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5738036f10 0x7f57380393d0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f57580a7de0 tx=0x7f5750010000 comp rx=0 tx=0).stop 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 10 monclient: shutdown 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 msgr2=0x7f57581cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f57577fe700 1 -- 172.21.15.5:0/2605061356 reap_dead start 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 secure :-1 s=READY pgs=6783 cs=0 l=1 rev1=1 crypto rx=0x7f574800c4e0 tx=0x7f574800c8a0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 shutdown_connections 2024-09-16T10:54:11.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 --2- 172.21.15.5:0/2605061356 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f575809f780 0x7f57581cabe0 unknown :-1 s=CLOSED pgs=6783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 >> 172.21.15.5:0/2605061356 conn(0x7f57580a53e0 msgr2=0x7f575806e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:11.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 shutdown_connections 2024-09-16T10:54:11.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 -- 172.21.15.5:0/2605061356 wait complete. 2024-09-16T10:54:11.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.205+0000 7f5757fff700 1 librados: shutdown 2024-09-16T10:54:11.250 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:54:11.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T10:54:11.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:54:11.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: get_monmap_and_config 2024-09-16T10:54:11.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: build_initial_monmap 2024-09-16T10:54:11.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: monmap: 2024-09-16T10:54:11.375 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:11.375 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.375 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:54:11.373811+0000 2024-09-16T10:54:11.375 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:54:11.373811+0000 2024-09-16T10:54:11.375 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:11.376 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.376 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:54:11.376 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:54:11.376 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:54:11.376 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:54:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:54:11.377 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 Processor -- start 2024-09-16T10:54:11.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 -- start start 2024-09-16T10:54:11.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: init 2024-09-16T10:54:11.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f512809e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f512809d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f512809dfd0 0x7f512813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-a con 0x7f512809dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128070e40 con 0x7f512809d6a0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512809fef0 con 0x7f512809e8e0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51280a01d0 con 0x7f512809dfd0 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512e2ef700 10 monclient(hunting): authenticate will time out at 1109.951220s 2024-09-16T10:54:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f512809dfd0 0x7f512813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512caec700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f512809d6a0 0x7f5128138b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37240/0 (socket says 172.21.15.5:37240) 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): get_auth_request con 0x7f512809dfd0 auth_method 0 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512caec700 1 -- 172.21.15.5:0/1007304724 learned_addr learned my addr 172.21.15.5:0/1007304724 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.369+0000 7f512daee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 10 monclient(hunting): handle_auth_done global_id 28545 payload 274 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 -- 172.21.15.5:0/1007304724 >> v1:172.21.15.143:6789/0 conn(0x7f512809e8e0 legacy=0x7f512809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 -- 172.21.15.5:0/1007304724 >> v1:172.21.15.5:6789/0 conn(0x7f512809d6a0 legacy=0x7f5128138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 10 monclient: _finish_hunting 0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 monclient: found mon.noname-a 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 1 -- 172.21.15.5:0/1007304724 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337752297 0 0) 0x7f5128070e40 con 0x7f512809d6a0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 -- 172.21.15.5:0/1007304724 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512809f9f0 con 0x7f512809dfd0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 --2- 172.21.15.5:0/1007304724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809dfd0 0x7f512813c2b0 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 crypto rx=0x7f5120009dc0 tx=0x7f512000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d89e69c5adf8129 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 1 -- 172.21.15.5:0/1007304724 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512000e070 con 0x7f512809dfd0 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: dump: 2024-09-16T10:54:11.397 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: _finish_auth 0 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: _check_auth_tickets 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 5 monclient: authenticate success, global_id 28545 2024-09-16T10:54:11.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 1 -- 172.21.15.5:0/1007304724 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512000bb90 con 0x7f512809dfd0 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: get_monmap_and_config success 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 4 set_mon_vals no callback set 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 1 -- 172.21.15.5:0/1007304724 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5120012430 con 0x7f512809dfd0 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f5127fff700 10 monclient: dump: 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.399 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: shutdown 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- 172.21.15.5:0/1007304724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809dfd0 msgr2=0x7f512813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1007304724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809dfd0 0x7f512813c2b0 secure :-1 s=READY pgs=6787 cs=0 l=1 rev1=1 crypto rx=0x7f5120009dc0 tx=0x7f512000b330 comp rx=0 tx=0).stop 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- 172.21.15.5:0/1007304724 shutdown_connections 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1007304724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809dfd0 0x7f512813c2b0 unknown :-1 s=CLOSED pgs=6787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- 172.21.15.5:0/1007304724 >> 172.21.15.5:0/1007304724 conn(0x7f51280a45d0 msgr2=0x7f51280a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:11.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- 172.21.15.5:0/1007304724 shutdown_connections 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- 172.21.15.5:0/1007304724 wait complete. 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: build_initial_monmap 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: monmap: 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:54:11.401 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 librados: starting msgr at 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 librados: starting objecter 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 Processor -- start 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- start start 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 librados: setting wanted keys 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 librados: calling monclient init 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: init 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: _reopen_session rank -1 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 0x7f51281c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-a con 0x7f512809d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f512809dfd0 0x7f5128133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-b con 0x7f512809dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512809e8e0 0x7f51281ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): picked mon.noname-c con 0x7f512809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5128120a80 con 0x7f512809d6a0 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512809b8e0 con 0x7f512809dfd0 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512809b0d0 con 0x7f512809e8e0 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): _renew_subs 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512e2ef700 10 monclient(hunting): authenticate will time out at 1109.955072s 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512809e8e0 msgr2=0x7f51281ca4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512daee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512809e8e0 0x7f51281ca4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:54:11.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 0x7f51281c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 0x7f51281c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47626/0 (socket says 172.21.15.5:47626) 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 1 -- 172.21.15.5:0/1806257445 learned_addr learned my addr 172.21.15.5:0/1806257445 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): get_auth_request con 0x7f512809d6a0 auth_method 0 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f512809dfd0 0x7f5128133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): get_auth_request con 0x7f512809dfd0 auth_method 0 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.373+0000 7f512d2ed700 10 monclient(hunting): handle_auth_done global_id 28551 payload 824 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512809e8e0 msgr2=0x7f51281ca4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f512809e8e0 0x7f51281ca4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f512809dfd0 msgr2=0x7f5128133030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f512809dfd0 0x7f5128133030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 10 monclient: _finish_hunting 0 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 monclient: found mon.noname-a 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:54:11.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5120009a70 con 0x7f512809d6a0 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 client.?.objecter ms_handle_connect 0x7f512809d6a0 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512d2ed700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 0x7f51281c6f90 secure :-1 s=READY pgs=6788 cs=0 l=1 rev1=1 crypto rx=0x7f511800c500 tx=0x7f511800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51180052d0 con 0x7f512809d6a0 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: dump: 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.406 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: _finish_auth 0 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5118005430 con 0x7f512809d6a0 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 5 monclient: authenticate success, global_id 28551 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 monclient: _renew_subs 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5128120150 con 0x7f512809d6a0 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51180055a0 con 0x7f512809d6a0 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 monclient: dump: 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.408 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5126ffd700 4 set_mon_vals no callback set 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 client.28551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 monclient: _renew_subs 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512811f960 con 0x7f512809d6a0 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 1 librados: init done 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 0x7f51281ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5128133950 con 0x7f5128133570 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f51180038e0 con 0x7f512809d6a0 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 0x7f51281ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient: get_auth_request con 0x7f5128133570 auth_method 0 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 --2- 172.21.15.5:0/1806257445 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5114036f10 0x7f51140393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6887+0+0 (secure 0 0 0) 0x7f51180477b0 con 0x7f512809d6a0 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 10 client.28551.objecter ms_dispatch 0x7f5128000fd0 osd_map(171..171 src has 1..171) v4 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 3 client.28551.objecter handle_osd_map got epochs [171,171] > 0 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 3 client.28551.objecter handle_osd_map decoding full epoch 171 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f5125ffb700 20 client.28551.objecter dump_active .. 0 homeless 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512daee700 1 --2- 172.21.15.5:0/1806257445 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5114036f10 0x7f51140393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512daee700 10 monclient: get_auth_request con 0x7f5114036f10 auth_method 0 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 10 monclient(hunting): handle_auth_done global_id 44179 payload 824 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.377+0000 7f512caec700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f512809f3d0 con 0x7f5128133570 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 client.28551.objecter ms_handle_connect 0x7f5128133570 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 client.28551.objecter resend_mon_ops 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f512caec700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 0x7f51281ca9f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f511c007fd0 tx=0x7f511c001910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 client.28551.objecter ms_handle_connect 0x7f5114036f10 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f512daee700 1 --2- 172.21.15.5:0/1806257445 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5114036f10 0x7f51140393d0 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f512809e3d0 tx=0x7f5120011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c017070 con 0x7f5128133570 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 monclient: dump: 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f511c012560 con 0x7f5128133570 2024-09-16T10:54:11.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:54:11.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:54:11.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 msgr2=0x7f51281ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 0x7f51281ca9f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f511c007fd0 tx=0x7f511c001910 comp rx=0 tx=0).stop 2024-09-16T10:54:11.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 msgr2=0x7f51281ca9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:54:11.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.381+0000 7f5125ffb700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5128133570 0x7f51281ca9f0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f511c007fd0 tx=0x7f511c001910 comp rx=0 tx=0).stop 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512e2ef700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 0x7f51280a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512e2ef700 10 monclient(hunting): start opening mon connection 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5128067380 con 0x7f51281caf30 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 0x7f51280a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient: get_auth_request con 0x7f51281caf30 auth_method 0 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:54:11.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 10 monclient(hunting): handle_auth_done global_id 44182 payload 824 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 1 -- 172.21.15.5:0/1806257445 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f51281cb340 con 0x7f51281caf30 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f512caec700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 0x7f51280a1dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f511c016000 tx=0x7f511c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 client.28551.objecter ms_handle_connect 0x7f51281caf30 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 client.28551.objecter resend_mon_ops 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c023070 con 0x7f51281caf30 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:54:11.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 monclient: dump: 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4434 (secure 0 0 0) 0x7f511c01e110 con 0x7f51281caf30 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:54:11.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 msgr2=0x7f51280a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 0x7f51280a1dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f511c016000 tx=0x7f511c000ed0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 msgr2=0x7f51280a1dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.393+0000 7f5125ffb700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51281caf30 0x7f51280a1dc0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f511c016000 tx=0x7f511c000ed0 comp rx=0 tx=0).stop 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 28, 2024-09-16T10:54:11.418 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.419 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.420 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:54:11.421 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:54:11.422 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.423 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:54:11.424 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.425 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.426 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:54:11.427 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:54:11.428 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:54:11.429 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:54:11.430 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:54:11.430 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:54:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 10 librados: watch_flush enter 2024-09-16T10:54:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 10 librados: watch_flush exit 2024-09-16T10:54:11.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 20 client.28551.objecter shutdown clearing up homeless session... 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 10 client.28551.objecter successfully canceled tick 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5114036f10 msgr2=0x7f51140393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1806257445 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5114036f10 0x7f51140393d0 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f512809e3d0 tx=0x7f5120011000 comp rx=0 tx=0).stop 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 10 monclient: shutdown 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 msgr2=0x7f51281c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 --2- 172.21.15.5:0/1806257445 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f512809d6a0 0x7f51281c6f90 secure :-1 s=READY pgs=6788 cs=0 l=1 rev1=1 crypto rx=0x7f511800c500 tx=0x7f511800c810 comp rx=0 tx=0).stop 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512daee700 1 -- 172.21.15.5:0/1806257445 reap_dead start 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512daee700 1 -- 172.21.15.5:0/1806257445 reap_dead start 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 shutdown_connections 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 >> 172.21.15.5:0/1806257445 conn(0x7f51280a45d0 msgr2=0x7f5128133f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 shutdown_connections 2024-09-16T10:54:11.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 -- 172.21.15.5:0/1806257445 wait complete. 2024-09-16T10:54:11.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:54:11.397+0000 7f512e2ef700 1 librados: shutdown 2024-09-16T10:54:11.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:54:11.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:54:49.756 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7044 ms) 2024-09-16T10:54:49.756 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T10:54:49.756 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6958 ms) 2024-09-16T10:54:49.757 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T10:54:49.757 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7057 ms) 2024-09-16T10:54:49.757 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T10:54:49.757 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6965 ms) 2024-09-16T10:54:49.757 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T10:54:49.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6994 ms) 2024-09-16T10:54:49.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T10:54:49.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (10002 ms) 2024-09-16T10:54:49.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T10:54:49.758 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7043 ms) 2024-09-16T10:54:49.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T10:54:49.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6972 ms) 2024-09-16T10:54:49.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T10:54:49.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6990 ms) 2024-09-16T10:54:49.759 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T10:54:49.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7158 ms) 2024-09-16T10:54:49.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T10:54:49.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (21846 ms) 2024-09-16T10:54:49.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T10:54:49.760 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7023 ms) 2024-09-16T10:54:49.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T10:54:49.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6979 ms) 2024-09-16T10:54:49.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T10:54:49.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7033 ms) 2024-09-16T10:54:49.761 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T10:54:49.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7004 ms) 2024-09-16T10:54:49.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T10:54:49.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6985 ms) 2024-09-16T10:54:49.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (130054 ms total) 2024-09-16T10:54:49.762 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: 2024-09-16T10:54:49.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T10:54:49.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (221630 ms total) 2024-09-16T10:54:49.763 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T10:54:49.766 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.766 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24809 2024-09-16T10:54:49.766 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24809 2024-09-16T10:54:49.767 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.767 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24426 2024-09-16T10:54:49.767 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24426 2024-09-16T10:54:49.767 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.767 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24795 2024-09-16T10:54:49.768 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24795 2024-09-16T10:54:49.768 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.768 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24734 2024-09-16T10:54:49.768 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24734 2024-09-16T10:54:49.768 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24386 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24386 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24534 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24534 2024-09-16T10:54:49.769 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24626 2024-09-16T10:54:49.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24626 2024-09-16T10:54:49.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T10:54:49.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24513 2024-09-16T10:54:49.770 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24513 2024-09-16T10:54:52.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T10:54:52.747 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: seed 24525 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (195 ms) 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10730 ms) 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (8 ms) 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10933 ms total) 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T10:54:52.748 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7000 ms) 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8043 ms) 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9974 ms) 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [4] 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [5,4] 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting some heads 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-16T10:54:52.749 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47015 ms) 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10008 ms) 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8005 ms) 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8055 ms) 2024-09-16T10:54:52.750 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7989 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9978 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9002 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10003 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13111 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (22889 ms) 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T10:54:52.751 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8112 ms) 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12920 ms) 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8062 ms) 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T10:54:52.752 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7922 ms) 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: ok, hit_set contains 187:602f83fe:::foo:head 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9081 ms) 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg_num = 32 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 0 ls 1726484108,0 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 1 ls 1726484108,0 2024-09-16T10:55:08.459 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 2 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 3 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 4 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 5 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 6 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 7 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 8 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 9 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 10 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 11 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 12 ls 1726484108,0 2024-09-16T10:55:08.460 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 13 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 14 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 15 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 16 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 17 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 18 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 19 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 20 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 21 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 22 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 23 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 24 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 25 ls 1726484108,0 2024-09-16T10:55:08.461 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 26 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 27 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 28 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 29 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 30 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg 31 ls 1726484108,0 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: pg_num = 32 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6cac518f:::0:head 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:02547ec2:::1:head 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f905c69b:::2:head 2024-09-16T10:55:08.462 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:cfc208b3:::3:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d83876eb:::4:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b29083e3:::5:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c4fdafeb:::6:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:5c6b0b28:::7:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:bd63b0f1:::8:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e960b815:::9:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:52ea6a34:::10:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:89d3ae78:::11:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:de5d7c5f:::12:head 2024-09-16T10:55:08.463 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:566253c9:::13:head 2024-09-16T10:55:08.464 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:62a1935d:::14:head 2024-09-16T10:55:08.464 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:863748b0:::15:head 2024-09-16T10:55:08.464 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3958e169:::16:head 2024-09-16T10:55:08.464 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4d4dabf9:::17:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:8391935d:::18:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:28883081:::19:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:69259c59:::20:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4bdb80b7:::21:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a11c5d71:::22:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:271af37b:::23:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:95b121be:::24:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:58d1031b:::25:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:0a050783:::26:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c709704c:::27:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:cbe56eaf:::28:head 2024-09-16T10:55:08.465 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:86b4b162:::29:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:70d89383:::30:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:dd450c7c:::31:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6d5729b1:::32:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c388f3fb:::33:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:56cfea31:::34:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9dbc1bf7:::35:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:40b74ccd:::36:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4d5aaf42:::37:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:920f362c:::38:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6cc53222:::39:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9cad833f:::40:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:1ea84d41:::41:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c4480ef6:::42:head 2024-09-16T10:55:08.466 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a694361e:::43:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d1bd33e9:::44:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ddc2cd5d:::45:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:2b782207:::46:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7b187fca:::47:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:90ecdf6f:::48:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a5ed95fe:::49:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ea0eaa55:::50:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f33ef17b:::51:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a0d1b2f6:::52:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:60c5229e:::53:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:edcbc575:::54:head 2024-09-16T10:55:08.467 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:102cf253:::55:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:efb7fb0b:::56:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:50d0a326:::57:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d4dc5daf:::58:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3a130462:::59:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ec87ed71:::60:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d5bc9454:::61:head 2024-09-16T10:55:08.468 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3ddfe313:::62:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7c2816b9:::63:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:47e00e4d:::64:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c6410c18:::65:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b48ed237:::66:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:cd63ad31:::67:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b179e92b:::68:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:0d9f741a:::69:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6d3352ae:::70:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c6d5c19e:::71:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:bc4729c3:::72:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:77e930b9:::73:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:0abeecfd:::74:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b7c37e15:::75:head 2024-09-16T10:55:08.469 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b6378398:::76:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:02bd68de:::77:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:cc795d2d:::78:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:630d4fea:::79:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e0d29ef5:::80:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:fd6f13d2:::81:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:606461d5:::82:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:eadbdc43:::83:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:8761d0bb:::84:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9ef0186f:::85:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e0d41294:::86:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:961de695:::87:head 2024-09-16T10:55:08.470 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:1423148f:::88:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:633a8fa2:::89:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a8653809:::90:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3dac8b33:::91:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:35aad435:::92:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f6dcc343:::93:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:dbbdad87:::94:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:1cb48ce0:::95:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:03cd461c:::96:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:17a4ea99:::97:head 2024-09-16T10:55:08.471 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9993c9a7:::98:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6394211c:::99:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:94c7ae57:::100:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6fdee5bb:::101:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9a477fd1:::102:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:eb850916:::103:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:affc56b9:::104:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b42dc814:::105:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f319f8f0:::106:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9a40b9de:::107:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:8b524f28:::108:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e3de589f:::109:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:90f90a5b:::110:head 2024-09-16T10:55:08.472 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a7b4f1d7:::111:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:af51766e:::112:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b6f90bd1:::113:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e0261208:::114:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c9569ef7:::115:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:61bebe50:::116:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:fe93412b:::117:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d3d38bee:::118:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3100ba0c:::119:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d0560ada:::120:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f0ea8b35:::121:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:766f231a:::122:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a07a2582:::123:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:bd7c6b3a:::124:head 2024-09-16T10:55:08.473 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:fb2ddaff:::125:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4408e1fe:::126:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ee1df7a7:::127:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c3002909:::128:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4f48ffa9:::129:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:edf38733:::130:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c08425c0:::131:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:5f902d98:::132:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:41ea2c93:::133:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:813cee13:::134:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:0131818d:::135:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:26ba5a85:::136:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:381b8a5a:::137:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:28797e47:::138:head 2024-09-16T10:55:08.474 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:bfca7f22:::139:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:36807075:::140:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:80b03975:::141:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:5c15709b:::142:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f39ea15e:::143:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ea992956:::144:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:48887b1c:::145:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:9f24a9dd:::146:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:987f100b:::147:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d2dd3581:::148:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7fed1808:::149:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c80b70e9:::150:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:85ed90f9:::151:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:36428b24:::152:head 2024-09-16T10:55:08.475 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d044c34a:::153:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7c18bf58:::154:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d1c21232:::155:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a7a3c575:::156:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:87da0633:::157:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d5ac3822:::158:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3f20522d:::159:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6ca26563:::160:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:532ce135:::161:head 2024-09-16T10:55:08.476 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:c78863e6:::162:head 2024-09-16T10:55:41.434 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T10:55:41.435 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T10:55:41.435 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T10:55:41.436 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T10:55:41.439 INFO:tasks.ceph.mon.c:Started 2024-09-16T10:55:41.439 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:55:41.439 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.543+0000 7f452a538700 10 monclient: get_monmap_and_config 2024-09-16T10:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.543+0000 7f452a538700 10 monclient: build_initial_monmap 2024-09-16T10:55:41.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.543+0000 7f452a538700 10 monclient: monmap: 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:41.548345+0000 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:41.548345+0000 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:41.550 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 Processor -- start 2024-09-16T10:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- start start 2024-09-16T10:55:41.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient: init 2024-09-16T10:55:41.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f452409db80 0x7f452409df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): picked mon.noname-a con 0x7f452409db80 addr v2:172.21.15.5:3300/0 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4524132230 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): picked mon.noname-b con 0x7f452409f9d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452409aee0 con 0x7f4524132230 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f452409e4a0 con 0x7f452409db80 2024-09-16T10:55:41.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452409ac00 con 0x7f452409f9d0 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient(hunting): authenticate will time out at 1200.125170s 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f452409db80 0x7f452409df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f452409db80 0x7f452409df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52836/0 (socket says 172.21.15.5:52836) 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 -- 172.21.15.5:0/3732761799 learned_addr learned my addr 172.21.15.5:0/3732761799 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): get_auth_request con 0x7f452409db80 auth_method 0 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:41.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 10 monclient(hunting): get_auth_request con 0x7f452409f9d0 auth_method 0 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 1 -- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f452413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 1 -- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f452413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4528ad5700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient(hunting): handle_auth_done global_id 28668 payload 274 2024-09-16T10:55:41.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 -- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f452413a110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 -- 172.21.15.5:0/3732761799 >> v1:172.21.15.143:6789/0 conn(0x7f4524132230 legacy=0x7f452409f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 monclient: found mon.noname-a 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 -- 172.21.15.5:0/3732761799 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452409bfd0 con 0x7f452409db80 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4523fff700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f452409db80 0x7f452409df60 secure :-1 s=READY pgs=6822 cs=0 l=1 rev1=1 crypto rx=0x7f4514009940 tx=0x7f4514009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74568d59eb8eb1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 1 -- 172.21.15.5:0/3732761799 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451400ba20 con 0x7f452409db80 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: dump: 2024-09-16T10:55:41.564 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: _finish_auth 0 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:41.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 1 -- 172.21.15.5:0/3732761799 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4514009e90 con 0x7f452409db80 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 1 -- 172.21.15.5:0/3732761799 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451400bda0 con 0x7f452409db80 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f4522ffd700 10 monclient: dump: 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.566 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 5 monclient: authenticate success, global_id 28668 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 4 set_mon_vals no callback set 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 10 monclient: shutdown 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:41.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f452409db80 msgr2=0x7f452409df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f452409db80 0x7f452409df60 secure :-1 s=READY pgs=6822 cs=0 l=1 rev1=1 crypto rx=0x7f4514009940 tx=0x7f4514009c50 comp rx=0 tx=0).stop 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- 172.21.15.5:0/3732761799 shutdown_connections 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f452413a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 --2- 172.21.15.5:0/3732761799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f452409db80 0x7f452409df60 unknown :-1 s=CLOSED pgs=6822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- 172.21.15.5:0/3732761799 >> 172.21.15.5:0/3732761799 conn(0x7f45240a0b30 msgr2=0x7f45240a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- 172.21.15.5:0/3732761799 shutdown_connections 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.547+0000 7f452a538700 1 -- 172.21.15.5:0/3732761799 wait complete. 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: build_initial_monmap 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: monmap: 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:41.568 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 librados: starting msgr at 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 librados: starting objecter 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 Processor -- start 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- start start 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 librados: setting wanted keys 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 librados: calling monclient init 2024-09-16T10:55:41.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: init 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f452409db80 0x7f4524071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): picked mon.noname-b con 0x7f452409db80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): picked mon.noname-c con 0x7f452409f9d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 0x7f45241309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): picked mon.noname-a con 0x7f4524132230 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452409b1b0 con 0x7f4524132230 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:41.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45240a2a70 con 0x7f452409db80 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45241c8a40 con 0x7f452409f9d0 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient(hunting): authenticate will time out at 1200.129967s 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:36806/0 (socket says 172.21.15.5:36806) 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 -- 172.21.15.5:0/1596531669 learned_addr learned my addr 172.21.15.5:0/1596531669 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 10 monclient(hunting): get_auth_request con 0x7f452409f9d0 auth_method 0 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 0x7f45241309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:41.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): get_auth_request con 0x7f4524132230 auth_method 0 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f452409db80 0x7f4524071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 -- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f45241304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 -- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f45241304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): get_auth_request con 0x7f452409db80 auth_method 0 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:41.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45237fe700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4523fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient(hunting): handle_auth_done global_id 28671 payload 824 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 -- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 msgr2=0x7f45241304b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 -- 172.21.15.5:0/1596531669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f452409db80 msgr2=0x7f4524071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f452409db80 0x7f4524071080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient: _finish_hunting 0 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 monclient: found mon.noname-a 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 -- 172.21.15.5:0/1596531669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009570 con 0x7f4524132230 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4528ad5700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 0x7f45241309f0 secure :-1 s=READY pgs=6823 cs=0 l=1 rev1=1 crypto rx=0x7f451c007d20 tx=0x7f451c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 client.?.objecter ms_handle_connect 0x7f4524132230 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c018070 con 0x7f4524132230 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: dump: 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:41.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: _finish_auth 0 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c007e70 con 0x7f4524132230 2024-09-16T10:55:41.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 5 monclient: authenticate success, global_id 28671 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _renew_subs 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c00ede0 con 0x7f4524132230 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452411fbb0 con 0x7f4524132230 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f45227fc700 4 set_mon_vals no callback set 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f4520ff9700 10 monclient: dump: 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:41.576 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 client.28671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _renew_subs 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452411f8d0 con 0x7f4524132230 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.551+0000 7f452a538700 1 librados: init done 2024-09-16T10:55:41.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f451c014420 con 0x7f4524132230 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f452a538700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f452a538700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f452a538700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452411e730 con 0x7f4524132230 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 1 --2- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 0x7f4510039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 5450+0+0 (secure 0 0 0) 0x7f451c047bc0 con 0x7f4524132230 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 10 client.28671.objecter ms_dispatch 0x7f4524000fd0 osd_map(261..261 src has 1..261) v4 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 3 client.28671.objecter handle_osd_map got epochs [261,261] > 0 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 3 client.28671.objecter handle_osd_map decoding full epoch 261 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 20 client.28671.objecter dump_active .. 0 homeless 2024-09-16T10:55:41.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4523fff700 1 --2- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 0x7f4510039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4523fff700 10 monclient: get_auth_request con 0x7f4510036f60 auth_method 0 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4520ff9700 10 client.28671.objecter ms_handle_connect 0x7f4510036f60 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.555+0000 7f4523fff700 1 --2- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 0x7f4510039420 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f4514008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.563+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f451c0146d0 con 0x7f4524132230 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.563+0000 7f4520ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:41.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.563+0000 7f4520ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:41.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.799+0000 7f452a538700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:41.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.799+0000 7f452a538700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:41.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.799+0000 7f452a538700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:41.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.799+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f452409c2e0 con 0x7f4524132230 2024-09-16T10:55:41.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f4520ff9700 1 -- 172.21.15.5:0/1596531669 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f451c0122e0 con 0x7f4524132230 2024-09-16T10:55:41.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f4520ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:41.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f4520ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:41.837 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 10 librados: watch_flush enter 2024-09-16T10:55:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 10 librados: watch_flush exit 2024-09-16T10:55:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 20 client.28671.objecter shutdown clearing up homeless session... 2024-09-16T10:55:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 10 client.28671.objecter successfully canceled tick 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 msgr2=0x7f4510039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 0x7f4510039420 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f4514008030 comp rx=0 tx=0).stop 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 10 monclient: shutdown 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 msgr2=0x7f45241309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 0x7f45241309f0 secure :-1 s=READY pgs=6823 cs=0 l=1 rev1=1 crypto rx=0x7f451c007d20 tx=0x7f451c00e3c0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 shutdown_connections 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4510036f60 0x7f4510039420 unknown :-1 s=CLOSED pgs=3770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4524132230 0x7f45241309f0 unknown :-1 s=CLOSED pgs=6823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f452409f9d0 0x7f45241304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 --2- 172.21.15.5:0/1596531669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f452409db80 0x7f4524071080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:41.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 >> 172.21.15.5:0/1596531669 conn(0x7f45240a0b30 msgr2=0x7f452409c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:41.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 shutdown_connections 2024-09-16T10:55:41.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 -- 172.21.15.5:0/1596531669 wait complete. 2024-09-16T10:55:41.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:41.835+0000 7f452a538700 1 librados: shutdown 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7462ddf6:::1o.RoundTripAppendPP (2947 ms) 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3007 ms) 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3072 ms) 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2996 ms) 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: setting pool EIO 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: max_success 97, min_failed 98 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3935 ms) 2024-09-16T10:55:44.785 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3048 ms) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (131629 ms total) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2981 ms) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9004 ms) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14981 ms) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3016 ms) 2024-09-16T10:55:44.786 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29982 ms total) 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3110 ms) 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3110 ms total) 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (28896 ms) 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6980 ms) 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T10:55:44.787 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7033 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7005 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3039 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6982 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6958 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7076 ms) 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T10:55:44.788 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6940 ms) 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6999 ms) 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7086 ms) 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6939 ms) 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6980 ms) 2024-09-16T10:55:44.789 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-09-16T10:55:44.857 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.971+0000 7f5925394700 10 monclient: get_monmap_and_config 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.971+0000 7f5925394700 10 monclient: build_initial_monmap 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.971+0000 7f5925394700 10 monclient: monmap: 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:44.976811+0000 2024-09-16T10:55:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:44.976811+0000 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:44.978 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 Processor -- start 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 -- start start 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient: init 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59200a1ad0 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59200a25c0 0x7f59200a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:44.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): picked mon.noname-c con 0x7f59200a25c0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): picked mon.noname-b con 0x7f592013aca0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592009de20 con 0x7f59200a1ad0 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592009ec60 con 0x7f59200a25c0 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592009fc70 con 0x7f592013aca0 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient(hunting): authenticate will time out at 1203.553586s 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591ffff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f59200a1ad0 0x7f59200a1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33366/0 (socket says 172.21.15.5:33366) 2024-09-16T10:55:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591ffff700 1 -- 172.21.15.5:0/2697480422 learned_addr learned my addr 172.21.15.5:0/2697480422 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 10 monclient(hunting): get_auth_request con 0x7f592013aca0 auth_method 0 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2697480422 >> v2:172.21.15.143:3300/0 conn(0x7f59200a25c0 0x7f59200a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): get_auth_request con 0x7f59200a25c0 auth_method 0 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 1 -- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 msgr2=0x7f592013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:55:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 1 -- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 msgr2=0x7f592013d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5924b93700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 1 -- 172.21.15.5:0/2697480422 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278514880 0 0) 0x7f592009de20 con 0x7f59200a1ad0 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient(hunting): _negotiate 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient(hunting): my global_id is 28686 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 1 -- 172.21.15.5:0/2697480422 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f590c0035d0 con 0x7f59200a1ad0 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient(hunting): handle_auth_done global_id 44326 payload 274 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 -- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 msgr2=0x7f592013d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:44.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 -- 172.21.15.5:0/2697480422 >> v1:172.21.15.5:6789/0 conn(0x7f59200a1ad0 legacy=0x7f59200a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 monclient: found mon.noname-c 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 -- 172.21.15.5:0/2697480422 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592009e020 con 0x7f59200a25c0 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59200a25c0 0x7f59200a0000 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5908009850 tx=0x7f5908009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5140fd8bbe89486a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 1 -- 172.21.15.5:0/2697480422 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908011830 con 0x7f59200a25c0 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: dump: 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:44.991 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: _finish_auth 0 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:44.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 5 monclient: authenticate success, global_id 44326 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 1 -- 172.21.15.5:0/2697480422 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908011990 con 0x7f59200a25c0 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f5925394700 4 set_mon_vals no callback set 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 1 -- 172.21.15.5:0/2697480422 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908011d40 con 0x7f59200a25c0 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.975+0000 7f591effd700 10 monclient: dump: 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:44.993 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: shutdown 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2697480422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59200a25c0 msgr2=0x7f59200a0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59200a25c0 0x7f59200a0000 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5908009850 tx=0x7f5908009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2697480422 shutdown_connections 2024-09-16T10:55:44.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f592013aca0 0x7f592013d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- 172.21.15.5:0/2697480422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59200a25c0 0x7f59200a0000 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2697480422 >> 172.21.15.5:0/2697480422 conn(0x7f59200a68a0 msgr2=0x7f59200a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2697480422 shutdown_connections 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2697480422 wait complete. 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: build_initial_monmap 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: monmap: 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:44.995 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 librados: starting msgr at 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 librados: starting objecter 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 Processor -- start 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- start start 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 librados: setting wanted keys 2024-09-16T10:55:44.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 librados: calling monclient init 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: init 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): picked mon.noname-c con 0x7f59200a1ad0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): picked mon.noname-a con 0x7f59200a25c0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592013aca0 0x7f5920120b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): picked mon.noname-b con 0x7f592013aca0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59201243a0 con 0x7f59200a25c0 2024-09-16T10:55:44.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592013cbf0 con 0x7f592013aca0 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200a5b20 con 0x7f59200a1ad0 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient(hunting): authenticate will time out at 1203.558382s 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57312/0 (socket says 172.21.15.5:57312) 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 -- 172.21.15.5:0/2246320221 learned_addr learned my addr 172.21.15.5:0/2246320221 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): get_auth_request con 0x7f59200a25c0 auth_method 0 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:44.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 10 monclient(hunting): get_auth_request con 0x7f59200a1ad0 auth_method 0 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592013aca0 0x7f5920120b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): get_auth_request con 0x7f592013aca0 auth_method 0 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 1 -- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 msgr2=0x7f59201ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:55:44.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 1 -- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 msgr2=0x7f59201ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591ffff700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5924b93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient(hunting): handle_auth_done global_id 28689 payload 824 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 -- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 msgr2=0x7f59201ca670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 -- 172.21.15.5:0/2246320221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592013aca0 msgr2=0x7f5920120b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592013aca0 0x7f5920120b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 monclient: found mon.noname-a 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 -- 172.21.15.5:0/2246320221 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080094b0 con 0x7f59200a25c0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591f7fe700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 secure :-1 s=READY pgs=6827 cs=0 l=1 rev1=1 crypto rx=0x7f59080090b0 tx=0x7f5908004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 client.?.objecter ms_handle_connect 0x7f59200a25c0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908004480 con 0x7f59200a25c0 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: dump: 2024-09-16T10:55:45.001 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: _finish_auth 0 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: _check_auth_tickets 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:45.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590801f070 con 0x7f59200a25c0 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 5 monclient: authenticate success, global_id 28689 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _renew_subs 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592009d2b0 con 0x7f59200a25c0 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908003900 con 0x7f59200a25c0 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591cff9700 10 monclient: dump: 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:45.003 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 client.28689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _renew_subs 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f591e7fc700 4 set_mon_vals no callback set 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59201240c0 con 0x7f59200a25c0 2024-09-16T10:55:45.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.979+0000 7f5925394700 1 librados: init done 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f5925394700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f5925394700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f5925394700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59201244e0 con 0x7f59200a25c0 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5908021030 con 0x7f59200a25c0 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 1 --2- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 0x7f5900039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(265..265 src has 1..265) v4 ==== 5301+0+0 (secure 0 0 0) 0x7f5908049110 con 0x7f59200a25c0 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 10 client.28689.objecter ms_dispatch 0x7f5920000fd0 osd_map(265..265 src has 1..265) v4 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 3 client.28689.objecter handle_osd_map got epochs [265,265] > 0 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 3 client.28689.objecter handle_osd_map decoding full epoch 265 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 20 client.28689.objecter dump_active .. 0 homeless 2024-09-16T10:55:45.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591ffff700 1 --2- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 0x7f5900039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591ffff700 10 monclient: get_auth_request con 0x7f5900036f60 auth_method 0 2024-09-16T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591cff9700 10 client.28689.objecter ms_handle_connect 0x7f5900036f60 2024-09-16T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.983+0000 7f591ffff700 1 --2- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 0x7f5900039420 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f5910001560 tx=0x7f5910009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.987+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f59201244e0 con 0x7f59200a25c0 2024-09-16T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.987+0000 7f591cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:45.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:44.987+0000 7f591cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:45.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f5925394700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:45.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f5925394700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:45.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f5925394700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:45.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59201244e0 con 0x7f59200a25c0 2024-09-16T10:55:45.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f591cff9700 1 -- 172.21.15.5:0/2246320221 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f5908015610 con 0x7f59200a25c0 2024-09-16T10:55:45.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f591cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:45.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.223+0000 7f591cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:45.229 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:45.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 10 librados: watch_flush enter 2024-09-16T10:55:45.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 10 librados: watch_flush exit 2024-09-16T10:55:45.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 20 client.28689.objecter shutdown clearing up homeless session... 2024-09-16T10:55:45.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 10 client.28689.objecter successfully canceled tick 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 msgr2=0x7f5900039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 0x7f5900039420 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f5910001560 tx=0x7f5910009040 comp rx=0 tx=0).stop 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 10 monclient: shutdown 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 msgr2=0x7f5920139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:45.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 secure :-1 s=READY pgs=6827 cs=0 l=1 rev1=1 crypto rx=0x7f59080090b0 tx=0x7f5908004770 comp rx=0 tx=0).stop 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 shutdown_connections 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f592013aca0 0x7f5920120b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5900036f60 0x7f5900039420 unknown :-1 s=CLOSED pgs=3773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f59200a25c0 0x7f5920139fd0 unknown :-1 s=CLOSED pgs=6827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 --2- 172.21.15.5:0/2246320221 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f59200a1ad0 0x7f59201ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 >> 172.21.15.5:0/2246320221 conn(0x7f59200a68a0 msgr2=0x7f59200a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 shutdown_connections 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 -- 172.21.15.5:0/2246320221 wait complete. 2024-09-16T10:55:45.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:45.227+0000 7f5925394700 1 librados: shutdown 2024-09-16T10:55:48.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.355+0000 7fc99343e700 10 monclient: get_monmap_and_config 2024-09-16T10:55:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.355+0000 7fc99343e700 10 monclient: build_initial_monmap 2024-09-16T10:55:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.355+0000 7fc99343e700 10 monclient: monmap: 2024-09-16T10:55:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:48.360003+0000 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:48.360003+0000 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:48.362 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 Processor -- start 2024-09-16T10:55:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- start start 2024-09-16T10:55:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient: init 2024-09-16T10:55:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc98c0a2bf0 0x7fc98c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-c con 0x7fc98c0a2bf0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc98c09f780 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-b con 0x7fc98c0a3510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc98c0a3c40 con 0x7fc98c09f780 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98c0a4300 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c09fda0 con 0x7fc98c0a3510 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient(hunting): authenticate will time out at 1206.937006s 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:56402/0 (socket says 172.21.15.5:56402) 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 -- 172.21.15.5:0/3076145822 learned_addr learned my addr 172.21.15.5:0/3076145822 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 10 monclient(hunting): get_auth_request con 0x7fc98c0a3510 auth_method 0 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 --2- 172.21.15.5:0/3076145822 >> v2:172.21.15.143:3300/0 conn(0x7fc98c0a2bf0 0x7fc98c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): get_auth_request con 0x7fc98c0a2bf0 auth_method 0 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 -- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 msgr2=0x7fc98c1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 -- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 msgr2=0x7fc98c1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:55:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc992c3d700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient(hunting): handle_auth_done global_id 44332 payload 274 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 -- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 msgr2=0x7fc98c1360c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 -- 172.21.15.5:0/3076145822 >> v1:172.21.15.143:6789/0 conn(0x7fc98c09f780 legacy=0x7fc98c1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient: _finish_hunting 0 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 monclient: found mon.noname-c 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 -- 172.21.15.5:0/3076145822 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c0a0080 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99243c700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c0a2fd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc988009850 tx=0x7fc988009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17ae73bb9da6c0a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 1 -- 172.21.15.5:0/3076145822 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988011830 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: dump: 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:48.370 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: _finish_auth 0 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: _check_auth_tickets 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 5 monclient: authenticate success, global_id 44332 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 1 -- 172.21.15.5:0/3076145822 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc988011990 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 4 set_mon_vals no callback set 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 1 -- 172.21.15.5:0/3076145822 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988011d20 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99143a700 10 monclient: dump: 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 10 monclient: shutdown 2024-09-16T10:55:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- 172.21.15.5:0/3076145822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 msgr2=0x7fc98c0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c0a2fd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc988009850 tx=0x7fc988009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- 172.21.15.5:0/3076145822 shutdown_connections 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c0a3510 0x7fc98c1360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 --2- 172.21.15.5:0/3076145822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c0a2fd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- 172.21.15.5:0/3076145822 >> 172.21.15.5:0/3076145822 conn(0x7fc98c0a53e0 msgr2=0x7fc98c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- 172.21.15.5:0/3076145822 shutdown_connections 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.359+0000 7fc99343e700 1 -- 172.21.15.5:0/3076145822 wait complete. 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: build_initial_monmap 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: monmap: 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 librados: starting msgr at 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 librados: starting objecter 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 Processor -- start 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- start start 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 librados: setting wanted keys 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 librados: calling monclient init 2024-09-16T10:55:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: init 2024-09-16T10:55:48.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:48.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:55:48.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-c con 0x7fc98c09f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c1cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-b con 0x7fc98c0a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): picked mon.noname-a con 0x7fc98c0a3510 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c0a7c60 con 0x7fc98c0a3510 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c1251b0 con 0x7fc98c0a2bf0 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c070830 con 0x7fc98c09f780 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:48.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient(hunting): authenticate will time out at 1206.941277s 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57328/0 (socket says 172.21.15.5:57328) 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 -- 172.21.15.5:0/1421587095 learned_addr learned my addr 172.21.15.5:0/1421587095 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): get_auth_request con 0x7fc98c0a3510 auth_method 0 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 10 monclient(hunting): get_auth_request con 0x7fc98c09f780 auth_method 0 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c1cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:48.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): get_auth_request con 0x7fc98c0a2bf0 auth_method 0 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 1 -- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 msgr2=0x7fc98c1cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 1 -- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 msgr2=0x7fc98c1cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99243c700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc991c3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient(hunting): handle_auth_done global_id 28692 payload 824 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 -- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 msgr2=0x7fc98c1cae50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 -- 172.21.15.5:0/1421587095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 msgr2=0x7fc98c1cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c1cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient: _finish_hunting 0 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 monclient: found mon.noname-a 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 -- 172.21.15.5:0/1421587095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9880094b0 con 0x7fc98c0a3510 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc992c3d700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 secure :-1 s=READY pgs=6829 cs=0 l=1 rev1=1 crypto rx=0x7fc984007b00 tx=0x7fc984007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 client.?.objecter ms_handle_connect 0x7fc98c0a3510 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400da30 con 0x7fc98c0a3510 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: dump: 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.384 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: _finish_auth 0 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 5 monclient: authenticate success, global_id 28692 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _renew_subs 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:48.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98400f040 con 0x7fc98c0a3510 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98c125850 con 0x7fc98c0a3510 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400ddd0 con 0x7fc98c0a3510 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc982ffd700 10 monclient: dump: 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:48.386 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc990c39700 4 set_mon_vals no callback set 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 client.28692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _renew_subs 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98c126a90 con 0x7fc98c0a3510 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.363+0000 7fc99343e700 1 librados: init done 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc98401e3b0 con 0x7fc98c0a3510 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99343e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99343e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:48.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99343e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 1 --2- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 0x7fc970039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0a2fd0 con 0x7fc98c0a3510 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 6076+0+0 (secure 0 0 0) 0x7fc984047e90 con 0x7fc98c0a3510 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 10 client.28692.objecter ms_dispatch 0x7fc98c000fd0 osd_map(268..268 src has 1..268) v4 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 3 client.28692.objecter handle_osd_map got epochs [268,268] > 0 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 3 client.28692.objecter handle_osd_map decoding full epoch 268 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 20 client.28692.objecter dump_active .. 0 homeless 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99243c700 1 --2- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 0x7fc970039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99243c700 10 monclient: get_auth_request con 0x7fc970036f60 auth_method 0 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc982ffd700 10 client.28692.objecter ms_handle_connect 0x7fc970036f60 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.367+0000 7fc99243c700 1 --2- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 0x7fc970039420 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7fc988000c00 tx=0x7fc988008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:48.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.371+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc98401a020 con 0x7fc98c0a3510 2024-09-16T10:55:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.371+0000 7fc982ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:48.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.371+0000 7fc982ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:48.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.603+0000 7fc99343e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:48.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.603+0000 7fc99343e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:48.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.603+0000 7fc99343e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:48.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.603+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc98c0721d0 con 0x7fc98c0a3510 2024-09-16T10:55:48.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc982ffd700 1 -- 172.21.15.5:0/1421587095 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fc98401ee60 con 0x7fc98c0a3510 2024-09-16T10:55:48.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc982ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:48.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc982ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:48.610 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 10 librados: watch_flush enter 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 10 librados: watch_flush exit 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 20 client.28692.objecter shutdown clearing up homeless session... 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 10 client.28692.objecter successfully canceled tick 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 msgr2=0x7fc970039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 0x7fc970039420 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7fc988000c00 tx=0x7fc988008030 comp rx=0 tx=0).stop 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 10 monclient: shutdown 2024-09-16T10:55:48.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 msgr2=0x7fc98c123170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 secure :-1 s=READY pgs=6829 cs=0 l=1 rev1=1 crypto rx=0x7fc984007b00 tx=0x7fc984007e10 comp rx=0 tx=0).stop 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 shutdown_connections 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc970036f60 0x7fc970039420 unknown :-1 s=CLOSED pgs=3774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc98c0a3510 0x7fc98c123170 unknown :-1 s=CLOSED pgs=6829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc98c0a2bf0 0x7fc98c1cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 --2- 172.21.15.5:0/1421587095 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc98c09f780 0x7fc98c1cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 >> 172.21.15.5:0/1421587095 conn(0x7fc98c0a53e0 msgr2=0x7fc98c06ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 shutdown_connections 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 -- 172.21.15.5:0/1421587095 wait complete. 2024-09-16T10:55:48.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:48.607+0000 7fc99343e700 1 librados: shutdown 2024-09-16T10:55:51.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: get_monmap_and_config 2024-09-16T10:55:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: build_initial_monmap 2024-09-16T10:55:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: monmap: 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:51.745192+0000 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:51.745192+0000 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:51.746 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 Processor -- start 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- start start 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: init 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fb40b63f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 0x7f0fb40be850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-c con 0x7f0fb40234f0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0fb40bed90 0x7f0fb40c1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-a con 0x7f0fb40bed90 addr v2:172.21.15.5:3300/0 2024-09-16T10:55:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40b6030 con 0x7f0fb40b63f0 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb4020000 con 0x7f0fb40bed90 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb401fce0 con 0x7f0fb40234f0 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient(hunting): authenticate will time out at 1210.321641s 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1006700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0fb40b63f0 0x7f0fb40b67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33372/0 (socket says 172.21.15.5:33372) 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1006700 1 -- 172.21.15.5:0/2603448058 learned_addr learned my addr 172.21.15.5:0/2603448058 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2603448058 >> v2:172.21.15.5:3300/0 conn(0x7f0fb40bed90 0x7f0fb40c1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): get_auth_request con 0x7f0fb40bed90 auth_method 0 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 1 --2- 172.21.15.5:0/2603448058 >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 0x7f0fb40be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 10 monclient(hunting): get_auth_request con 0x7f0fb40234f0 auth_method 0 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 1 -- 172.21.15.5:0/2603448058 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610754117 0 0) 0x7f0fb40b6030 con 0x7f0fb40b63f0 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient(hunting): _negotiate 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient(hunting): my global_id is 28698 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 1 -- 172.21.15.5:0/2603448058 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa80035d0 con 0x7f0fb40b63f0 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient(hunting): handle_auth_done global_id 28695 payload 274 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 -- 172.21.15.5:0/2603448058 >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 msgr2=0x7f0fb40be850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2603448058 >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 0x7f0fb40be850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 -- 172.21.15.5:0/2603448058 >> v1:172.21.15.5:6789/0 conn(0x7f0fb40b63f0 legacy=0x7f0fb40b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient: _finish_hunting 0 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 monclient: found mon.noname-a 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:51.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 -- 172.21.15.5:0/2603448058 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4023940 con 0x7f0fb40bed90 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc0805700 1 --2- 172.21.15.5:0/2603448058 >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 0x7f0fb40be850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2603448058 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40bed90 0x7f0fb40c1180 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009850 tx=0x7f0fb0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d4f60a977a47b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 1 -- 172.21.15.5:0/2603448058 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0011830 con 0x7f0fb40bed90 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: dump: 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:51.754 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: _finish_auth 0 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 1 -- 172.21.15.5:0/2603448058 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0011990 con 0x7f0fb40bed90 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 5 monclient: authenticate success, global_id 28695 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 4 set_mon_vals no callback set 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 1 -- 172.21.15.5:0/2603448058 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0011d20 con 0x7f0fb40bed90 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0faffff700 10 monclient: dump: 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 10 monclient: shutdown 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- 172.21.15.5:0/2603448058 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40bed90 msgr2=0x7f0fb40c1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2603448058 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40bed90 0x7f0fb40c1180 secure :-1 s=READY pgs=6832 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0009850 tx=0x7f0fb0009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- 172.21.15.5:0/2603448058 shutdown_connections 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2603448058 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40bed90 0x7f0fb40c1180 unknown :-1 s=CLOSED pgs=6832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2603448058 >> v2:172.21.15.143:3300/0 conn(0x7f0fb40234f0 0x7f0fb40be850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.743+0000 7f0fc2008700 1 -- 172.21.15.5:0/2603448058 >> 172.21.15.5:0/2603448058 conn(0x7f0fb40064b0 msgr2=0x7f0fb40068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- 172.21.15.5:0/2603448058 shutdown_connections 2024-09-16T10:55:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- 172.21.15.5:0/2603448058 wait complete. 2024-09-16T10:55:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient: build_initial_monmap 2024-09-16T10:55:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient: monmap: 2024-09-16T10:55:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 librados: starting msgr at 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 librados: starting objecter 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 Processor -- start 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- start start 2024-09-16T10:55:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 librados: setting wanted keys 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 librados: calling monclient init 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient: init 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb40234f0 0x7f0fb40b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-b con 0x7f0fb40234f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 0x7f0fb40bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-a con 0x7f0fb40b63f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 0x7f0fb40bc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): picked mon.noname-c con 0x7f0fb40bed90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40a16a0 con 0x7f0fb40b63f0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40bea20 con 0x7f0fb40234f0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4020870 con 0x7f0fb40bed90 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc2008700 10 monclient(hunting): authenticate will time out at 1210.327402s 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 0x7f0fb40bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 0x7f0fb40bbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57350/0 (socket says 172.21.15.5:57350) 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1006700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb40234f0 0x7f0fb40b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 1 -- 172.21.15.5:0/2516663820 learned_addr learned my addr 172.21.15.5:0/2516663820 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 0x7f0fb40bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 10 monclient(hunting): get_auth_request con 0x7f0fb40b63f0 auth_method 0 2024-09-16T10:55:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc0805700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1006700 10 monclient(hunting): get_auth_request con 0x7f0fb40234f0 auth_method 0 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1006700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1006700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1006700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): get_auth_request con 0x7f0fb40bed90 auth_method 0 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.747+0000 7f0fc1807700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc0805700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc0805700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc0805700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1006700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1006700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1006700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 10 monclient(hunting): handle_auth_done global_id 54101 payload 824 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 -- 172.21.15.5:0/2516663820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb40234f0 msgr2=0x7f0fb40b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb40234f0 0x7f0fb40b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 -- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 msgr2=0x7f0fb40bbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 0x7f0fb40bbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 10 monclient: _finish_hunting 0 2024-09-16T10:55:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 monclient: found mon.noname-c 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 -- 172.21.15.5:0/2516663820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00094b0 con 0x7f0fb40bed90 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1807700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 0x7f0fb40bc240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0016000 tx=0x7f0fb0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 client.?.objecter ms_handle_connect 0x7f0fb40bed90 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00044d0 con 0x7f0fb40bed90 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: dump: 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:51.768 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: _finish_auth 0 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: _check_auth_tickets 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb00036c0 con 0x7f0fb40bed90 2024-09-16T10:55:51.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 5 monclient: authenticate success, global_id 54101 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _renew_subs 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0003a70 con 0x7f0fb40bed90 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb40a10e0 con 0x7f0fb40bed90 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0faf7fe700 4 set_mon_vals no callback set 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 monclient: dump: 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 client.54101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _renew_subs 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40a01f0 con 0x7f0fb40bed90 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 1 librados: init done 2024-09-16T10:55:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0fb0021070 con 0x7f0fb40bed90 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 --2- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 0x7f0fa0039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5416+0+0 (secure 0 0 0) 0x7f0fb0048af0 con 0x7f0fb40bed90 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 10 client.54101.objecter ms_dispatch 0x7f0fbc000fd0 osd_map(271..271 src has 1..271) v4 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 3 client.54101.objecter handle_osd_map got epochs [271,271] > 0 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 3 client.54101.objecter handle_osd_map decoding full epoch 271 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fadffb700 20 client.54101.objecter dump_active .. 0 homeless 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1006700 1 --2- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 0x7f0fa0039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc1006700 10 monclient: get_auth_request con 0x7f0fa0036f90 auth_method 0 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.751+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb40b67d0 con 0x7f0fb40bed90 2024-09-16T10:55:51.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.755+0000 7f0fadffb700 10 client.54101.objecter ms_handle_connect 0x7f0fa0036f90 2024-09-16T10:55:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.755+0000 7f0fc1006700 1 --2- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 0x7f0fa0039450 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc04f480 tx=0x7f0fbc08c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.763+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0fb00239c0 con 0x7f0fb40bed90 2024-09-16T10:55:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.763+0000 7f0fadffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:51.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:51.763+0000 7f0fadffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:52.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:52.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:52.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0fb4008550 con 0x7f0fb40bed90 2024-09-16T10:55:52.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fadffb700 1 -- 172.21.15.5:0/2516663820 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f0fb0032080 con 0x7f0fb40bed90 2024-09-16T10:55:52.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fadffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:52.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fadffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:52.010 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 librados: watch_flush enter 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 librados: watch_flush exit 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 20 client.54101.objecter shutdown clearing up homeless session... 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 client.54101.objecter successfully canceled tick 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 msgr2=0x7f0fa0039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 0x7f0fa0039450 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc04f480 tx=0x7f0fbc08c040 comp rx=0 tx=0).stop 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 10 monclient: shutdown 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 msgr2=0x7f0fb40bc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 0x7f0fb40bc240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0016000 tx=0x7f0fb0004770 comp rx=0 tx=0).stop 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 shutdown_connections 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0fb40bed90 0x7f0fb40bc240 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0fa0036f90 0x7f0fa0039450 unknown :-1 s=CLOSED pgs=3775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0fb40b63f0 0x7f0fb40bbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 --2- 172.21.15.5:0/2516663820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb40234f0 0x7f0fb40b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 >> 172.21.15.5:0/2516663820 conn(0x7f0fb40064b0 msgr2=0x7f0fb401bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 shutdown_connections 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.007+0000 7f0fc2008700 1 -- 172.21.15.5:0/2516663820 wait complete. 2024-09-16T10:55:52.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.011+0000 7f0fc2008700 1 librados: shutdown 2024-09-16T10:55:52.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:55:52.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:55:52.024 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:55:52.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: get_monmap_and_config 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: monmap: 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:52.137472+0000 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:52.137472+0000 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:52.138 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:52.139 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:52.139 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 Processor -- start 2024-09-16T10:55:52.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 -- start start 2024-09-16T10:55:52.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: init 2024-09-16T10:55:52.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd82009b3e0 0x7fd82009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-c con 0x7fd82009b3e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd82009bd00 0x7fd82009c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-a con 0x7fd82009bd00 addr v2:172.21.15.5:3300/0 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd820132b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-b con 0x7fd8201326b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82009fd30 con 0x7fd82009bd00 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82009e350 con 0x7fd82009b3e0 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82009ca20 con 0x7fd8201326b0 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient(hunting): authenticate will time out at 1210.714087s 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd82009bd00 0x7fd82009c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd820132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd82009bd00 0x7fd82009c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57362/0 (socket says 172.21.15.5:57362) 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 -- 172.21.15.5:0/4216380489 learned_addr learned my addr 172.21.15.5:0/4216380489 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): get_auth_request con 0x7fd82009bd00 auth_method 0 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): get_auth_request con 0x7fd8201326b0 auth_method 0 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/4216380489 >> v2:172.21.15.143:3300/0 conn(0x7fd82009b3e0 0x7fd82009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request con 0x7fd82009b3e0 auth_method 0 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient(hunting): handle_auth_done global_id 28707 payload 274 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 -- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 msgr2=0x7fd820132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 --2- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd820132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 -- 172.21.15.5:0/4216380489 >> v2:172.21.15.143:3300/0 conn(0x7fd82009b3e0 msgr2=0x7fd82009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 --2- 172.21.15.5:0/4216380489 >> v2:172.21.15.143:3300/0 conn(0x7fd82009b3e0 0x7fd82009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 monclient: found mon.noname-a 2024-09-16T10:55:52.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 -- 172.21.15.5:0/4216380489 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82009cd00 con 0x7fd82009bd00 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81effd700 1 --2- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009bd00 0x7fd82009c110 secure :-1 s=READY pgs=6835 cs=0 l=1 rev1=1 crypto rx=0x7fd810009910 tx=0x7fd810009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7949073e6dd2d838 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 1 -- 172.21.15.5:0/4216380489 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810011830 con 0x7fd82009bd00 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: dump: 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.148 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 5 monclient: authenticate success, global_id 28707 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 1 -- 172.21.15.5:0/4216380489 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd810011990 con 0x7fd82009bd00 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:52.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 1 -- 172.21.15.5:0/4216380489 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810011d40 con 0x7fd82009bd00 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd824fd6700 4 set_mon_vals no callback set 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.135+0000 7fd81e7fc700 10 monclient: dump: 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.150 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: shutdown 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009bd00 msgr2=0x7fd82009c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009bd00 0x7fd82009c110 secure :-1 s=READY pgs=6835 cs=0 l=1 rev1=1 crypto rx=0x7fd810009910 tx=0x7fd810009c20 comp rx=0 tx=0).stop 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- 172.21.15.5:0/4216380489 shutdown_connections 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd820132b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- 172.21.15.5:0/4216380489 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009bd00 0x7fd82009c110 unknown :-1 s=CLOSED pgs=6835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- 172.21.15.5:0/4216380489 >> v2:172.21.15.143:3300/0 conn(0x7fd82009b3e0 0x7fd82009b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- 172.21.15.5:0/4216380489 >> 172.21.15.5:0/4216380489 conn(0x7fd8200a45d0 msgr2=0x7fd8200a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- 172.21.15.5:0/4216380489 shutdown_connections 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- 172.21.15.5:0/4216380489 wait complete. 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: monmap: 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 librados: starting msgr at 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 librados: starting objecter 2024-09-16T10:55:52.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 Processor -- start 2024-09-16T10:55:52.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- start start 2024-09-16T10:55:52.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 librados: setting wanted keys 2024-09-16T10:55:52.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 librados: calling monclient init 2024-09-16T10:55:52.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: init 2024-09-16T10:55:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:55:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd8201caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-a con 0x7fd82009b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd8201cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-b con 0x7fd82009bd00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): picked mon.noname-c con 0x7fd8201326b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd820070830 con 0x7fd82009b3e0 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82011bd50 con 0x7fd82009bd00 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82011c650 con 0x7fd8201326b0 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd824fd6700 10 monclient(hunting): authenticate will time out at 1210.719147s 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd8201caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd8201caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57372/0 (socket says 172.21.15.5:57372) 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 1 -- 172.21.15.5:0/1761866216 learned_addr learned my addr 172.21.15.5:0/1761866216 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request con 0x7fd82009b3e0 auth_method 0 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): get_auth_request con 0x7fd8201326b0 auth_method 0 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd8201cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): get_auth_request con 0x7fd82009bd00 auth_method 0 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.139+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): handle_auth_done global_id 54113 payload 824 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 msgr2=0x7fd8201cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd8201cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 msgr2=0x7fd8201caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd8201caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 monclient: found mon.noname-c 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810009570 con 0x7fd8201326b0 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd8201caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd8201cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd81800efc0 tx=0x7fd81800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 client.?.objecter ms_handle_connect 0x7fd8201326b0 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81800cc90 con 0x7fd8201326b0 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 msgr2=0x7fd82011a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 msgr2=0x7fd82011a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd81800efc0 tx=0x7fd81800c3a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:52.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: dump: 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd81800efc0 tx=0x7fd81800c3a0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 msgr2=0x7fd82011a530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd8201326b0 0x7fd82011a530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd80c007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): picked mon.a con 0x7fd82009b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd80c009b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): picked mon.b con 0x7fd82009bd00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:52.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 0x7fd80c00c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): picked mon.c con 0x7fd80c00a550 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c00d490 con 0x7fd82009b3e0 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c00d5d0 con 0x7fd82009bd00 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c00d740 con 0x7fd80c00a550 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd80c007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request con 0x7fd82009b3e0 auth_method 0 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 0x7fd80c00c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): get_auth_request con 0x7fd80c00a550 auth_method 0 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd80c009b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): get_auth_request con 0x7fd82009bd00 auth_method 0 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81effd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient(hunting): handle_auth_done global_id 54113 payload 293 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 msgr2=0x7fd80c009b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd82009bd00 0x7fd80c009b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 msgr2=0x7fd80c007140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd80c007140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 monclient: found mon.c 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fd80c00d970 con 0x7fd80c00a550 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 5 monclient: authenticate success, global_id 54113 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd82009b3e0 0x7fd80c007140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 monclient: _renew_subs 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82011dde0 con 0x7fd80c00a550 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 client.54113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:52.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 monclient: _renew_subs 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82011d820 con 0x7fd80c00a550 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 1 librados: init done 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 client.54113.objecter ms_handle_connect 0x7fd80c00a550 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81ffff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 0x7fd80c00c9e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd81800ef90 tx=0x7fd81800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 client.54113.objecter resend_mon_ops 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818023070 con 0x7fd80c00a550 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: dump: 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.176 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81800f040 con 0x7fd80c00a550 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81dffb700 4 set_mon_vals no callback set 2024-09-16T10:55:52.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 0x7fd800004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd818003a40 con 0x7fd80c00a550 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800007d30 con 0x7fd800002850 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 0x7fd800004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient: get_auth_request con 0x7fd800002850 auth_method 0 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.143+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd8201326b0 0x7fd80c0416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.2 v2:172.21.15.5:3301/0 4 ==== osd_map(272..272 src has 1..272) v4 ==== 5309+0+0 (secure 0 0 0) 0x7fd818049350 con 0x7fd80c00a550 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_done global_id 28713 payload 824 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 client.54113.objecter ms_dispatch 0x7fd820000fd0 osd_map(272..272 src has 1..272) v4 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 3 client.54113.objecter handle_osd_map got epochs [272,272] > 0 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 3 client.54113.objecter handle_osd_map decoding full epoch 272 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81f7fe700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd800007ef0 con 0x7fd800002850 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81effd700 1 --2- 172.21.15.5:0/1761866216 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd8201326b0 0x7fd80c0416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 20 client.54113.objecter dump_active .. 0 homeless 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 0x7fd800004cf0 secure :-1 s=READY pgs=6836 cs=0 l=1 rev1=1 crypto rx=0x7fd814012560 tx=0x7fd814012640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 client.54113.objecter ms_handle_connect 0x7fd800002850 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81effd700 10 monclient: get_auth_request con 0x7fd8201326b0 auth_method 0 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 client.54113.objecter resend_mon_ops 2024-09-16T10:55:52.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814004030 con 0x7fd800002850 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 monclient: dump: 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.180 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd814004190 con 0x7fd800002850 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 msgr2=0x7fd800004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 0x7fd800004cf0 secure :-1 s=READY pgs=6836 cs=0 l=1 rev1=1 crypto rx=0x7fd814012560 tx=0x7fd814012640 comp rx=0 tx=0).stop 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 msgr2=0x7fd800004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800002850 0x7fd800004cf0 secure :-1 s=CLOSED pgs=6836 cs=0 l=1 rev1=1 crypto rx=0x7fd814012560 tx=0x7fd814012640 comp rx=0 tx=0).stop 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd807fff700 10 client.54113.objecter ms_handle_connect 0x7fd8201326b0 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.147+0000 7fd81effd700 1 --2- 172.21.15.5:0/1761866216 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd8201326b0 0x7fd80c0416d0 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7fd810009c60 tx=0x7fd810008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:52.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 0x7fd8000071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000a1d0 con 0x7fd800006d70 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 0x7fd8000071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient: get_auth_request con 0x7fd800006d70 auth_method 0 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 10 monclient(hunting): handle_auth_done global_id 28716 payload 824 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 1 -- 172.21.15.5:0/1761866216 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd80000a390 con 0x7fd800006d70 2024-09-16T10:55:52.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 client.54113.objecter ms_handle_connect 0x7fd800006d70 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 client.54113.objecter resend_mon_ops 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81f7fe700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 0x7fd8000071e0 secure :-1 s=READY pgs=6837 cs=0 l=1 rev1=1 crypto rx=0x7fd814009640 tx=0x7fd8140047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814005c40 con 0x7fd800006d70 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 monclient: dump: 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.183 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7fd814005da0 con 0x7fd800006d70 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 msgr2=0x7fd8000071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 0x7fd8000071e0 secure :-1 s=READY pgs=6837 cs=0 l=1 rev1=1 crypto rx=0x7fd814009640 tx=0x7fd8140047e0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 msgr2=0x7fd8000071e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd807fff700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd800006d70 0x7fd8000071e0 secure :-1 s=CLOSED pgs=6837 cs=0 l=1 rev1=1 crypto rx=0x7fd814009640 tx=0x7fd8140047e0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.184 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.185 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:52.186 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:52.187 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:52.188 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.189 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.190 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:52.191 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.192 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.193 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.194 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.195 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.196 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.197 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.198 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:52.199 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:55:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.200 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.201 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:52.201 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:52.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 librados: watch_flush enter 2024-09-16T10:55:52.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 librados: watch_flush exit 2024-09-16T10:55:52.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 20 client.54113.objecter shutdown clearing up homeless session... 2024-09-16T10:55:52.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 client.54113.objecter successfully canceled tick 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd8201326b0 msgr2=0x7fd80c0416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 --2- 172.21.15.5:0/1761866216 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd8201326b0 0x7fd80c0416d0 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7fd810009c60 tx=0x7fd810008030 comp rx=0 tx=0).stop 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 10 monclient: shutdown 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 msgr2=0x7fd80c00c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 0x7fd80c00c9e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd81800ef90 tx=0x7fd81800b040 comp rx=0 tx=0).stop 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd81ffff700 1 -- 172.21.15.5:0/1761866216 reap_dead start 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 shutdown_connections 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 --2- 172.21.15.5:0/1761866216 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd80c00a550 0x7fd80c00c9e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 >> 172.21.15.5:0/1761866216 conn(0x7fd8200a45d0 msgr2=0x7fd820134b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 shutdown_connections 2024-09-16T10:55:52.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 -- 172.21.15.5:0/1761866216 wait complete. 2024-09-16T10:55:52.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.155+0000 7fd824fd6700 1 librados: shutdown 2024-09-16T10:55:52.205 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:52.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:55:52.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.323+0000 7f0c89ba3700 10 monclient: get_monmap_and_config 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.323+0000 7f0c89ba3700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.323+0000 7f0c89ba3700 10 monclient: monmap: 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:52.328609+0000 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:52.328609+0000 2024-09-16T10:55:52.329 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:52.330 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 Processor -- start 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- start start 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient: init 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c8409e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:52.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c8409d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c8413c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-b con 0x7f0c8409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c84070e40 con 0x7f0c8409e8e0 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8409fef0 con 0x7f0c8409d6a0 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840a01d0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient(hunting): authenticate will time out at 1210.904978s 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c88ba1700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0c8409e8e0 0x7f0c8409ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33388/0 (socket says 172.21.15.5:33388) 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c88ba1700 1 -- 172.21.15.5:0/1844147793 learned_addr learned my addr 172.21.15.5:0/1844147793 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 --2- 172.21.15.5:0/1844147793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c8413c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): get_auth_request con 0x7f0c8409dfd0 auth_method 0 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 1 -- 172.21.15.5:0/1844147793 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229077588 0 0) 0x7f0c84070e40 con 0x7f0c8409e8e0 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient(hunting): _negotiate 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient(hunting): my global_id is 28719 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 1 -- 172.21.15.5:0/1844147793 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c0035d0 con 0x7f0c8409e8e0 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient(hunting): handle_auth_done global_id 54116 payload 274 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 -- 172.21.15.5:0/1844147793 >> v1:172.21.15.143:6789/0 conn(0x7f0c8409d6a0 legacy=0x7f0c84138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 -- 172.21.15.5:0/1844147793 >> v1:172.21.15.5:6789/0 conn(0x7f0c8409e8e0 legacy=0x7f0c8409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 monclient: found mon.noname-b 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 -- 172.21.15.5:0/1844147793 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8409f9f0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c893a2700 1 --2- 172.21.15.5:0/1844147793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c8413c2b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009df0 tx=0x7f0c7c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2b279563bb1bf1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 1 -- 172.21.15.5:0/1844147793 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c00e070 con 0x7f0c8409dfd0 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: dump: 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.344 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 1 -- 172.21.15.5:0/1844147793 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c00bae0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 5 monclient: authenticate success, global_id 54116 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:52.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 1 -- 172.21.15.5:0/1844147793 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c00be90 con 0x7f0c8409dfd0 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 4 set_mon_vals no callback set 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c7b7fe700 10 monclient: dump: 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.346 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 10 monclient: shutdown 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- 172.21.15.5:0/1844147793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 msgr2=0x7f0c8413c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/1844147793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c8413c2b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009df0 tx=0x7f0c7c00b330 comp rx=0 tx=0).stop 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- 172.21.15.5:0/1844147793 shutdown_connections 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/1844147793 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c8413c2b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- 172.21.15.5:0/1844147793 >> 172.21.15.5:0/1844147793 conn(0x7f0c840a45d0 msgr2=0x7f0c840a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- 172.21.15.5:0/1844147793 shutdown_connections 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.327+0000 7f0c89ba3700 1 -- 172.21.15.5:0/1844147793 wait complete. 2024-09-16T10:55:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: monmap: 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 librados: starting msgr at 2024-09-16T10:55:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 librados: starting objecter 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 Processor -- start 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- start start 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 librados: setting wanted keys 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 librados: calling monclient init 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: init 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c8409d6a0 0x7f0c84122330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-b con 0x7f0c8409d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c84122870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-c con 0x7f0c8409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8409e8e0 0x7f0c84122db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): picked mon.noname-a con 0x7f0c8409e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84126420 con 0x7f0c8409e8e0 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8406f7f0 con 0x7f0c8409d6a0 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8409b0d0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient(hunting): authenticate will time out at 1210.909825s 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c84122870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8409e8e0 0x7f0c84122db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c84122870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:56452/0 (socket says 172.21.15.5:56452) 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 -- 172.21.15.5:0/279999259 learned_addr learned my addr 172.21.15.5:0/279999259 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): get_auth_request con 0x7f0c8409dfd0 auth_method 0 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): get_auth_request con 0x7f0c8409e8e0 auth_method 0 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c8409d6a0 0x7f0c84122330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c893a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 10 monclient(hunting): get_auth_request con 0x7f0c8409d6a0 auth_method 0 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient(hunting): handle_auth_done global_id 54119 payload 824 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c8409d6a0 msgr2=0x7f0c84122330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c8409d6a0 0x7f0c84122330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8409e8e0 msgr2=0x7f0c84122db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8409e8e0 0x7f0c84122db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 monclient: found mon.noname-c 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c009a70 con 0x7f0c8409dfd0 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c8409d6a0 0x7f0c84122330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7bfff700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c84122870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c530 tx=0x7f0c7400c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 client.?.objecter ms_handle_connect 0x7f0c8409dfd0 2024-09-16T10:55:52.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740052d0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: dump: 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.354 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c74005430 con 0x7f0c8409dfd0 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 5 monclient: authenticate success, global_id 54119 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _renew_subs 2024-09-16T10:55:52.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740055a0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8406cc40 con 0x7f0c8409dfd0 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c7affd700 4 set_mon_vals no callback set 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c797fa700 10 monclient: dump: 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.356 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 client.54119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _renew_subs 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c84123c30 con 0x7f0c8409dfd0 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.331+0000 7f0c89ba3700 1 librados: init done 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c89ba3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84067380 con 0x7f0c840a1ec0 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient: get_auth_request con 0x7f0c840a1ec0 auth_method 0 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0c740038e0 con 0x7f0c8409dfd0 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_done global_id 54122 payload 824 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c8409ad80 con 0x7f0c840a1ec0 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 1 --2- 172.21.15.5:0/279999259 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0c70036f60 0x7f0c70039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 msgr2=0x7f0c840a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 msgr2=0x7f0c840a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c8000ea90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 5309+0+0 (secure 0 0 0) 0x7f0c74047270 con 0x7f0c8409dfd0 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 10 client.54119.objecter ms_dispatch 0x7f0c84000fd0 osd_map(272..272 src has 1..272) v4 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 3 client.54119.objecter handle_osd_map got epochs [272,272] > 0 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 3 client.54119.objecter handle_osd_map decoding full epoch 272 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 20 client.54119.objecter dump_active .. 0 homeless 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c893a2700 1 --2- 172.21.15.5:0/279999259 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0c70036f60 0x7f0c70039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c893a2700 10 monclient: get_auth_request con 0x7f0c70036f60 auth_method 0 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c893a2700 1 --2- 172.21.15.5:0/279999259 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0c70036f60 0x7f0c70039420 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009df0 tx=0x7f0c7c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.335+0000 7f0c797fa700 10 client.54119.objecter ms_handle_connect 0x7f0c70036f60 2024-09-16T10:55:52.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: get_auth_request con 0x7f0c840a1ec0 auth_method 0 2024-09-16T10:55:52.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: _init_auth method 2 2024-09-16T10:55:52.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: _init_auth already have auth, reseting 2024-09-16T10:55:52.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: handle_auth_reply_more payload 9 2024-09-16T10:55:52.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:52.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 10 monclient: handle_auth_done global_id 54122 payload 293 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 client.54119.objecter ms_handle_connect 0x7f0c840a1ec0 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 client.54119.objecter resend_mon_ops 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80005960 con 0x7f0c840a1ec0 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 monclient: dump: 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f0c80005ac0 con 0x7f0c840a1ec0 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:52.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 msgr2=0x7f0c840a22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80009040 comp rx=0 tx=0).stop 2024-09-16T10:55:52.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 msgr2=0x7f0c840a22a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.539+0000 7f0c797fa700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a1ec0 0x7f0c840a22a0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80009040 comp rx=0 tx=0).stop 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.551+0000 7f0c89ba3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.551+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 0x7f0c841d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.551+0000 7f0c89ba3700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.551+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84125f60 con 0x7f0c840a27e0 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.551+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 0x7f0c841d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient: get_auth_request con 0x7f0c840a27e0 auth_method 0 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 10 monclient(hunting): handle_auth_done global_id 54125 payload 824 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 1 -- 172.21.15.5:0/279999259 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0c84126110 con 0x7f0c840a27e0 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c88ba1700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 0x7f0c841d49d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 client.54119.objecter ms_handle_connect 0x7f0c840a27e0 2024-09-16T10:55:52.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 client.54119.objecter resend_mon_ops 2024-09-16T10:55:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800072d0 con 0x7f0c840a27e0 2024-09-16T10:55:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 monclient: dump: 2024-09-16T10:55:52.560 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f0c80028070 con 0x7f0c840a27e0 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 msgr2=0x7f0c841d49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 0x7f0c841d49d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80000ed0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 msgr2=0x7f0c841d49d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c797fa700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c840a27e0 0x7f0c841d49d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c80000f60 tx=0x7f0c80000ed0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:52.563 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 1, 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.564 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:52.566 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.567 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.568 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:52.571 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.573 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:52.574 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.576 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.577 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:52.578 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 10 librados: watch_flush enter 2024-09-16T10:55:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 10 librados: watch_flush exit 2024-09-16T10:55:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 20 client.54119.objecter shutdown clearing up homeless session... 2024-09-16T10:55:52.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 10 client.54119.objecter successfully canceled tick 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0c70036f60 msgr2=0x7f0c70039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/279999259 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0c70036f60 0x7f0c70039420 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009df0 tx=0x7f0c7c011000 comp rx=0 tx=0).stop 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 10 monclient: shutdown 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 msgr2=0x7f0c84122870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 --2- 172.21.15.5:0/279999259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0c8409dfd0 0x7f0c84122870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c530 tx=0x7f0c7400c840 comp rx=0 tx=0).stop 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c893a2700 1 -- 172.21.15.5:0/279999259 reap_dead start 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c893a2700 1 -- 172.21.15.5:0/279999259 reap_dead start 2024-09-16T10:55:52.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 shutdown_connections 2024-09-16T10:55:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 >> 172.21.15.5:0/279999259 conn(0x7f0c840a45d0 msgr2=0x7f0c84133d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 shutdown_connections 2024-09-16T10:55:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 -- 172.21.15.5:0/279999259 wait complete. 2024-09-16T10:55:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.555+0000 7f0c89ba3700 1 librados: shutdown 2024-09-16T10:55:52.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:52.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:55:52.582 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:55:52.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: get_monmap_and_config 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: monmap: 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:52.705321+0000 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:52.705321+0000 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.706 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:52.707 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:52.707 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:52.707 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:52.707 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:52.707 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 Processor -- start 2024-09-16T10:55:52.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 -- start start 2024-09-16T10:55:52.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: init 2024-09-16T10:55:52.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7f409b3e0 0x7fd7f409b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-c con 0x7fd7f409b3e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7f409bd00 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7f409c110 0x7fd7f4132dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-a con 0x7fd7f409c110 addr v2:172.21.15.5:3300/0 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f409c560 con 0x7fd7f409bd00 2024-09-16T10:55:52.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f409e350 con 0x7fd7f409c110 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f409ca20 con 0x7fd7f409b3e0 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient(hunting): authenticate will time out at 1211.282082s 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7f409c110 0x7fd7f4132dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7f409c110 0x7fd7f4132dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59874/0 (socket says 172.21.15.5:59874) 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 -- 172.21.15.5:0/1216080738 learned_addr learned my addr 172.21.15.5:0/1216080738 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): get_auth_request con 0x7fd7f409c110 auth_method 0 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f895c700 1 --2- 172.21.15.5:0/1216080738 >> v2:172.21.15.143:3300/0 conn(0x7fd7f409b3e0 0x7fd7f409b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f895c700 10 monclient(hunting): get_auth_request con 0x7fd7f409b3e0 auth_method 0 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f895c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f895c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f895c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient(hunting): handle_auth_done global_id 28725 payload 274 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 -- 172.21.15.5:0/1216080738 >> v2:172.21.15.143:3300/0 conn(0x7fd7f409b3e0 msgr2=0x7fd7f409b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 --2- 172.21.15.5:0/1216080738 >> v2:172.21.15.143:3300/0 conn(0x7fd7f409b3e0 0x7fd7f409b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 -- 172.21.15.5:0/1216080738 >> v1:172.21.15.143:6789/0 conn(0x7fd7f409bd00 legacy=0x7fd7f41326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 monclient: found mon.noname-a 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 -- 172.21.15.5:0/1216080738 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f409cd00 con 0x7fd7f409c110 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f915d700 1 --2- 172.21.15.5:0/1216080738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409c110 0x7fd7f4132dc0 secure :-1 s=READY pgs=6839 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009e50 tx=0x7fd7ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2fc19995e23045 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 1 -- 172.21.15.5:0/1216080738 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec00e070 con 0x7fd7f409c110 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: dump: 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 5 monclient: authenticate success, global_id 28725 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 1 -- 172.21.15.5:0/1216080738 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec00b980 con 0x7fd7f409c110 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 4 set_mon_vals no callback set 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 1 -- 172.21.15.5:0/1216080738 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec00bd30 con 0x7fd7f409c110 2024-09-16T10:55:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f37fe700 10 monclient: dump: 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 10 monclient: shutdown 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 -- 172.21.15.5:0/1216080738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409c110 msgr2=0x7fd7f4132dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.703+0000 7fd7f995e700 1 --2- 172.21.15.5:0/1216080738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409c110 0x7fd7f4132dc0 secure :-1 s=READY pgs=6839 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009e50 tx=0x7fd7ec00b330 comp rx=0 tx=0).stop 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- 172.21.15.5:0/1216080738 shutdown_connections 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 --2- 172.21.15.5:0/1216080738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409c110 0x7fd7f4132dc0 unknown :-1 s=CLOSED pgs=6839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 --2- 172.21.15.5:0/1216080738 >> v2:172.21.15.143:3300/0 conn(0x7fd7f409b3e0 0x7fd7f409b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- 172.21.15.5:0/1216080738 >> 172.21.15.5:0/1216080738 conn(0x7fd7f40a45d0 msgr2=0x7fd7f40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- 172.21.15.5:0/1216080738 shutdown_connections 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- 172.21.15.5:0/1216080738 wait complete. 2024-09-16T10:55:52.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: monmap: 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.727 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:52.728 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:52.728 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:52.728 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:52.728 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 librados: starting msgr at 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 librados: starting objecter 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 Processor -- start 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- start start 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 librados: setting wanted keys 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 librados: calling monclient init 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: init 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409b3e0 0x7fd7f41c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-a con 0x7fd7f409b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:52.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f409bd00 0x7fd7f41c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-b con 0x7fd7f409bd00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): picked mon.noname-c con 0x7fd7f409c110 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40709b0 con 0x7fd7f409b3e0 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f41c7f60 con 0x7fd7f409bd00 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f409b0d0 con 0x7fd7f409c110 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient(hunting): authenticate will time out at 1211.286497s 2024-09-16T10:55:52.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409b3e0 0x7fd7f41c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44288/0 (socket says 172.21.15.5:44288) 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 -- 172.21.15.5:0/3403434787 learned_addr learned my addr 172.21.15.5:0/3403434787 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): get_auth_request con 0x7fd7f409c110 auth_method 0 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): get_auth_request con 0x7fd7f409b3e0 auth_method 0 2024-09-16T10:55:52.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f409bd00 0x7fd7f41c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): get_auth_request con 0x7fd7f409bd00 auth_method 0 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient(hunting): handle_auth_done global_id 54128 payload 824 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f409bd00 msgr2=0x7fd7f41c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f409bd00 0x7fd7f41c4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409b3e0 msgr2=0x7fd7f41c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7f409b3e0 0x7fd7f41c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 monclient: found mon.noname-c 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec009b30 con 0x7fd7f409c110 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f915d700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b730 tx=0x7fd7ec004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 client.?.objecter ms_handle_connect 0x7fd7f409c110 2024-09-16T10:55:52.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec004830 con 0x7fd7f409c110 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: dump: 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.736 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 5 monclient: authenticate success, global_id 54128 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec0092e0 con 0x7fd7f409c110 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: _renew_subs 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f41a9400 con 0x7fd7f409c110 2024-09-16T10:55:52.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec004990 con 0x7fd7f409c110 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f995e700 10 client.54128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.707+0000 7fd7f17fa700 10 monclient: dump: 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.739 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 10 monclient: _renew_subs 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f41ab640 con 0x7fd7f409c110 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 1 librados: init done 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f2ffd700 4 set_mon_vals no callback set 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 0x7fd7f402ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f4067380 con 0x7fd7f402b6a0 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 0x7fd7f402ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient: get_auth_request con 0x7fd7f402b6a0 auth_method 0 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd7ec01c680 con 0x7fd7f409c110 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 10 monclient(hunting): handle_auth_done global_id 44377 payload 824 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7f409ff00 con 0x7fd7f402b6a0 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 --2- 172.21.15.5:0/3403434787 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7d8036f60 0x7fd7d8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 5309+0+0 (secure 0 0 0) 0x7fd7ec0493e0 con 0x7fd7f409c110 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 client.54128.objecter ms_dispatch 0x7fd7f4000fd0 osd_map(272..272 src has 1..272) v4 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 3 client.54128.objecter handle_osd_map got epochs [272,272] > 0 2024-09-16T10:55:52.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 3 client.54128.objecter handle_osd_map decoding full epoch 272 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f895c700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 0x7fd7f402ba80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4011830 tx=0x7fd7e4011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 20 client.54128.objecter dump_active .. 0 homeless 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f3fff700 1 --2- 172.21.15.5:0/3403434787 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7d8036f60 0x7fd7d8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 client.54128.objecter ms_handle_connect 0x7fd7f402b6a0 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 client.54128.objecter resend_mon_ops 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f3fff700 10 monclient: get_auth_request con 0x7fd7d8036f60 auth_method 0 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004550 con 0x7fd7f402b6a0 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 monclient: dump: 2024-09-16T10:55:52.744 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f3fff700 1 --2- 172.21.15.5:0/3403434787 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7d8036f60 0x7fd7d8039420 secure :-1 s=READY pgs=3779 cs=0 l=1 rev1=1 crypto rx=0x7fd7f409f3f0 tx=0x7fd7dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 client.54128.objecter ms_handle_connect 0x7fd7d8036f60 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd7e40036c0 con 0x7fd7f402b6a0 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 msgr2=0x7fd7f402ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 0x7fd7f402ba80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4011830 tx=0x7fd7e4011bf0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 msgr2=0x7fd7f402ba80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.711+0000 7fd7f17fa700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402b6a0 0x7fd7f402ba80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f995e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f995e700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 0x7fd7f402c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f995e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f41a4e10 con 0x7fd7f402bfc0 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 0x7fd7f402c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient: get_auth_request con 0x7fd7f402bfc0 auth_method 0 2024-09-16T10:55:52.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.723+0000 7fd7f895c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f895c700 10 monclient(hunting): handle_auth_done global_id 44380 payload 824 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f895c700 1 -- 172.21.15.5:0/3403434787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd7f409b7c0 con 0x7fd7f402bfc0 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f895c700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 0x7fd7f402c3d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4011830 tx=0x7fd7e4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 client.54128.objecter ms_handle_connect 0x7fd7f402bfc0 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 client.54128.objecter resend_mon_ops 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4025070 con 0x7fd7f402bfc0 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 monclient: dump: 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7fd7e40202e0 con 0x7fd7f402bfc0 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 msgr2=0x7fd7f402c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 0x7fd7f402c3d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4011830 tx=0x7fd7e4004770 comp rx=0 tx=0).stop 2024-09-16T10:55:52.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 msgr2=0x7fd7f402c3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:52.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f17fa700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7f402bfc0 0x7fd7f402c3d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.752 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:52.753 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 1, 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.754 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.755 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.756 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.757 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.758 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:52.759 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:52.760 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:52.761 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:52.762 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:52.763 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:52.764 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.765 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:52.766 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.767 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:52.768 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.769 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:52.770 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.771 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.772 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:52.773 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:52.774 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:52.775 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:52.775 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:52.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 10 librados: watch_flush enter 2024-09-16T10:55:52.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 10 librados: watch_flush exit 2024-09-16T10:55:52.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 20 client.54128.objecter shutdown clearing up homeless session... 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 10 client.54128.objecter successfully canceled tick 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7d8036f60 msgr2=0x7fd7d8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 --2- 172.21.15.5:0/3403434787 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd7d8036f60 0x7fd7d8039420 secure :-1 s=READY pgs=3779 cs=0 l=1 rev1=1 crypto rx=0x7fd7f409f3f0 tx=0x7fd7dc009040 comp rx=0 tx=0).stop 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 10 monclient: shutdown 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 msgr2=0x7fd7f41ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f915d700 1 -- 172.21.15.5:0/3403434787 reap_dead start 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00b730 tx=0x7fd7ec004af0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 shutdown_connections 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 --2- 172.21.15.5:0/3403434787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd7f409c110 0x7fd7f41ca530 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 >> 172.21.15.5:0/3403434787 conn(0x7fd7f40a45d0 msgr2=0x7fd7f40a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 shutdown_connections 2024-09-16T10:55:52.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 -- 172.21.15.5:0/3403434787 wait complete. 2024-09-16T10:55:52.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.727+0000 7fd7f995e700 1 librados: shutdown 2024-09-16T10:55:52.778 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:52.778 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:55:52.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:55:52.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: get_monmap_and_config 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: monmap: 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:52.894342+0000 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:52.894342+0000 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:52.896 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:52.896 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 Processor -- start 2024-09-16T10:55:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 -- start start 2024-09-16T10:55:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: init 2024-09-16T10:55:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f049809e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 0x7f0498138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-b con 0x7f049809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f049809dfd0 0x7f049813b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-c con 0x7f049809dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0498070e40 con 0x7f049809e8e0 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980a01d0 con 0x7f049809dfd0 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049809fef0 con 0x7f049809d6a0 2024-09-16T10:55:52.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0497fff700 10 monclient(hunting): authenticate will time out at 1211.470835s 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 0x7f0498138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0496ffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f049809e8e0 0x7f049809ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36396/0 (socket says 172.21.15.5:36396) 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f0496ffd700 1 -- 172.21.15.5:0/2733816789 learned_addr learned my addr 172.21.15.5:0/2733816789 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): get_auth_request con 0x7f049809d6a0 auth_method 0 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 1 --2- 172.21.15.5:0/2733816789 >> v2:172.21.15.143:3300/0 conn(0x7f049809dfd0 0x7f049813b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): get_auth_request con 0x7f049809dfd0 auth_method 0 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 1 -- 172.21.15.5:0/2733816789 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984544617 0 0) 0x7f0498070e40 con 0x7f049809e8e0 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 10 monclient(hunting): _negotiate 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 10 monclient(hunting): my global_id is 28731 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04967fc700 1 -- 172.21.15.5:0/2733816789 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04840035d0 con 0x7f049809e8e0 2024-09-16T10:55:52.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f04977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient(hunting): handle_auth_done global_id 54131 payload 274 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 -- 172.21.15.5:0/2733816789 >> v2:172.21.15.143:3300/0 conn(0x7f049809dfd0 msgr2=0x7f049813b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 --2- 172.21.15.5:0/2733816789 >> v2:172.21.15.143:3300/0 conn(0x7f049809dfd0 0x7f049813b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 -- 172.21.15.5:0/2733816789 >> v1:172.21.15.5:6789/0 conn(0x7f049809e8e0 legacy=0x7f049809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 monclient: found mon.noname-b 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.891+0000 7f048ffff700 1 -- 172.21.15.5:0/2733816789 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049809f9f0 con 0x7f049809d6a0 2024-09-16T10:55:52.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f048ffff700 1 --2- 172.21.15.5:0/2733816789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 0x7f0498138b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0480009850 tx=0x7f0480009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f95ead23881b587 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 1 -- 172.21.15.5:0/2733816789 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480011830 con 0x7f049809d6a0 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: dump: 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 1 -- 172.21.15.5:0/2733816789 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0480011990 con 0x7f049809d6a0 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 5 monclient: authenticate success, global_id 54131 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 1 -- 172.21.15.5:0/2733816789 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480011d40 con 0x7f049809d6a0 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f04967fc700 10 monclient: dump: 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 4 set_mon_vals no callback set 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 10 monclient: shutdown 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 -- 172.21.15.5:0/2733816789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 msgr2=0x7f0498138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 --2- 172.21.15.5:0/2733816789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 0x7f0498138b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0480009850 tx=0x7f0480009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 -- 172.21.15.5:0/2733816789 shutdown_connections 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 --2- 172.21.15.5:0/2733816789 >> v2:172.21.15.143:3300/0 conn(0x7f049809dfd0 0x7f049813b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 --2- 172.21.15.5:0/2733816789 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809d6a0 0x7f0498138b80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 -- 172.21.15.5:0/2733816789 >> 172.21.15.5:0/2733816789 conn(0x7f04980a45d0 msgr2=0x7f04980a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 -- 172.21.15.5:0/2733816789 shutdown_connections 2024-09-16T10:55:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 1 -- 172.21.15.5:0/2733816789 wait complete. 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 10 monclient: build_initial_monmap 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.895+0000 7f0497fff700 10 monclient: monmap: 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:52.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 librados: starting msgr at 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 librados: starting objecter 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 Processor -- start 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- start start 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 librados: setting wanted keys 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 librados: calling monclient init 2024-09-16T10:55:52.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: init 2024-09-16T10:55:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:52.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-a con 0x7f049809d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809dfd0 0x7f0498133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-c con 0x7f049809dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 0x7f04981340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): picked mon.noname-b con 0x7f049809e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04981c91f0 con 0x7f049809d6a0 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049809b0d0 con 0x7f049809e8e0 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049806cba0 con 0x7f049809dfd0 2024-09-16T10:55:52.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient(hunting): authenticate will time out at 1211.477739s 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59884/0 (socket says 172.21.15.5:59884) 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 -- 172.21.15.5:0/4163680357 learned_addr learned my addr 172.21.15.5:0/4163680357 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): get_auth_request con 0x7f049809d6a0 auth_method 0 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809dfd0 0x7f0498133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): get_auth_request con 0x7f049809dfd0 auth_method 0 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 0x7f04981340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 10 monclient(hunting): get_auth_request con 0x7f049809e8e0 auth_method 0 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:52.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f048ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient(hunting): handle_auth_done global_id 28734 payload 824 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 -- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809dfd0 msgr2=0x7f0498133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809dfd0 0x7f0498133b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 -- 172.21.15.5:0/4163680357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 msgr2=0x7f04981340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 0x7f04981340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 monclient: found mon.noname-a 2024-09-16T10:55:52.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 -- 172.21.15.5:0/4163680357 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800094b0 con 0x7f049809d6a0 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f04977fe700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 0x7f04981340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0496ffd700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 secure :-1 s=READY pgs=6841 cs=0 l=1 rev1=1 crypto rx=0x7f048800c730 tx=0x7f048800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 client.?.objecter ms_handle_connect 0x7f049809d6a0 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488005d90 con 0x7f049809d6a0 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: dump: 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.920 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: _finish_auth 0 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 5 monclient: authenticate success, global_id 28734 2024-09-16T10:55:52.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04880070d0 con 0x7f049809d6a0 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _renew_subs 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0498121540 con 0x7f049809d6a0 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488007480 con 0x7f049809d6a0 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0495ffb700 4 set_mon_vals no callback set 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0494ff9700 10 monclient: dump: 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:52.922 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 client.28734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _renew_subs 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0498120ca0 con 0x7f049809d6a0 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.899+0000 7f0497fff700 1 librados: init done 2024-09-16T10:55:52.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f04880075e0 con 0x7f049809d6a0 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0497fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0497fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0497fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049809ecc0 con 0x7f049809d6a0 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 1 --2- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 0x7f0478039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 5566+0+0 (secure 0 0 0) 0x7f0488047480 con 0x7f049809d6a0 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 10 client.28734.objecter ms_dispatch 0x7f0498000fd0 osd_map(273..273 src has 1..273) v4 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 3 client.28734.objecter handle_osd_map got epochs [273,273] > 0 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 3 client.28734.objecter handle_osd_map decoding full epoch 273 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 20 client.28734.objecter dump_active .. 0 homeless 2024-09-16T10:55:52.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f048ffff700 1 --2- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 0x7f0478039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f048ffff700 10 monclient: get_auth_request con 0x7f0478036f60 auth_method 0 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f048ffff700 1 --2- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 0x7f0478039420 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f04800090b0 tx=0x7f0480008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.903+0000 7f0494ff9700 10 client.28734.objecter ms_handle_connect 0x7f0478036f60 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.911+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f048800e280 con 0x7f049809d6a0 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.911+0000 7f0494ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:52.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:52.911+0000 7f0494ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0497fff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:55:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0497fff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:55:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0497fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:53.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f049806f510 con 0x7f049809d6a0 2024-09-16T10:55:53.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0494ff9700 1 -- 172.21.15.5:0/4163680357 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0488019ab0 con 0x7f049809d6a0 2024-09-16T10:55:53.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0494ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:55:53.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f0494ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:53.153 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.153 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.147+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=3254335374,config=692619927,health=344733071,kv=564032378,logm=2636917743}) 2024-09-16T10:55:53.153 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.151+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=3254335374,config=692619927,health=3385433911,kv=564032378,logm=1216214519}) 2024-09-16T10:55:53.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.151+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.151+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=3254335374,config=692619927,health=344733071,kv=564032378,logm=2636917743}) 2024-09-16T10:55:53.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.151+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3254335374,config=692619927,health=3720241440,kv=564032378,logm=145646584}) 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525075255}) 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376246828}) 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525075255}) 2024-09-16T10:55:53.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509615278}) 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 10 librados: watch_flush enter 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 10 librados: watch_flush exit 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 20 client.28734.objecter shutdown clearing up homeless session... 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 10 client.28734.objecter successfully canceled tick 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 msgr2=0x7f0478039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:53.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 0x7f0478039420 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f04800090b0 tx=0x7f0480008030 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 10 monclient: shutdown 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 msgr2=0x7f04981c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 secure :-1 s=READY pgs=6841 cs=0 l=1 rev1=1 crypto rx=0x7f048800c730 tx=0x7f048800ca40 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 shutdown_connections 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049809e8e0 0x7f04981340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f049809dfd0 0x7f0498133b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0478036f60 0x7f0478039420 unknown :-1 s=CLOSED pgs=3780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 --2- 172.21.15.5:0/4163680357 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f049809d6a0 0x7f04981c8480 unknown :-1 s=CLOSED pgs=6841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 >> 172.21.15.5:0/4163680357 conn(0x7f04980a45d0 msgr2=0x7f04980a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 shutdown_connections 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 -- 172.21.15.5:0/4163680357 wait complete. 2024-09-16T10:55:53.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f0497fff700 1 librados: shutdown 2024-09-16T10:55:53.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666720682}) 2024-09-16T10:55:53.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4045783976}) 2024-09-16T10:55:53.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666720682}) 2024-09-16T10:55:53.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.155+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=697825839}) 2024-09-16T10:55:53.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336564217}) 2024-09-16T10:55:53.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=592648076}) 2024-09-16T10:55:53.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336564217}) 2024-09-16T10:55:53.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.159+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3692100818}) 2024-09-16T10:55:53.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954829085}) 2024-09-16T10:55:53.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417572414}) 2024-09-16T10:55:53.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954829085}) 2024-09-16T10:55:53.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898087427}) 2024-09-16T10:55:53.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726515694}) 2024-09-16T10:55:53.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064164929}) 2024-09-16T10:55:53.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726515694}) 2024-09-16T10:55:53.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.163+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354866615}) 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764854714}) 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169549305}) 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764854714}) 2024-09-16T10:55:53.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024867408}) 2024-09-16T10:55:53.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861459211}) 2024-09-16T10:55:53.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686806033}) 2024-09-16T10:55:53.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861459211}) 2024-09-16T10:55:53.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208600020}) 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451121581}) 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977602961}) 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451121581}) 2024-09-16T10:55:53.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.167+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131317280}) 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886789509}) 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710654330}) 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886789509}) 2024-09-16T10:55:53.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711052531}) 2024-09-16T10:55:53.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883947421}) 2024-09-16T10:55:53.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053278033}) 2024-09-16T10:55:53.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883947421}) 2024-09-16T10:55:53.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.171+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124968792}) 2024-09-16T10:55:53.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985615477}) 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533712103}) 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985615477}) 2024-09-16T10:55:53.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403045934}) 2024-09-16T10:55:53.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160722536}) 2024-09-16T10:55:53.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=963211483}) 2024-09-16T10:55:53.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160722536}) 2024-09-16T10:55:53.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.175+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2056475241}) 2024-09-16T10:55:53.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724736441}) 2024-09-16T10:55:53.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2680955787}) 2024-09-16T10:55:53.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724736441}) 2024-09-16T10:55:53.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886714503}) 2024-09-16T10:55:53.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393629162}) 2024-09-16T10:55:53.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567317782}) 2024-09-16T10:55:53.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393629162}) 2024-09-16T10:55:53.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249781338}) 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.179+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936988332}) 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132793365}) 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936988332}) 2024-09-16T10:55:53.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914424943}) 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686118188}) 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071060644}) 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686118188}) 2024-09-16T10:55:53.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1753726651}) 2024-09-16T10:55:53.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3098408746}) 2024-09-16T10:55:53.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163718454}) 2024-09-16T10:55:53.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3098408746}) 2024-09-16T10:55:53.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=217761605}) 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086660833}) 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390918231}) 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086660833}) 2024-09-16T10:55:53.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.183+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=388693098}) 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870778405}) 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421597104}) 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870778405}) 2024-09-16T10:55:53.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000435780}) 2024-09-16T10:55:53.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957134059}) 2024-09-16T10:55:53.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828567595}) 2024-09-16T10:55:53.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957134059}) 2024-09-16T10:55:53.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=203514211}) 2024-09-16T10:55:53.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278875493}) 2024-09-16T10:55:53.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1435983804}) 2024-09-16T10:55:53.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278875493}) 2024-09-16T10:55:53.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.187+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232090786}) 2024-09-16T10:55:53.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726513284}) 2024-09-16T10:55:53.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943937155}) 2024-09-16T10:55:53.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=726513284}) 2024-09-16T10:55:53.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291403413}) 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384100403}) 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744031879}) 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384100403}) 2024-09-16T10:55:53.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1823492589}) 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946954952}) 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.191+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801462326}) 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946954952}) 2024-09-16T10:55:53.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2053678239}) 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985358853}) 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977017846}) 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985358853}) 2024-09-16T10:55:53.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269470749}) 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593183383}) 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925179372}) 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593183383}) 2024-09-16T10:55:53.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629398304}) 2024-09-16T10:55:53.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482588652}) 2024-09-16T10:55:53.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284824884}) 2024-09-16T10:55:53.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482588652}) 2024-09-16T10:55:53.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.195+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2637713963}) 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=1601536256,mdsmap=3585761405,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=236392167,mon_config_key=2146694628,monmap=1472519444,osd_metadata=3752038628,osd_pg_creating=3002402524,osd_snap=2585451761}) 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=1601536256,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1748900067,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3002402524,osd_snap=401517955}) 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=1601536256,mdsmap=3585761405,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=236392167,mon_config_key=2146694628,monmap=1472519444,osd_metadata=3752038628,osd_pg_creating=3002402524,osd_snap=2585451761}) 2024-09-16T10:55:53.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=2398684397,mdsmap=1531151339,mgr=121140300,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=4261559810,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3002402524,osd_snap=430064531}) 2024-09-16T10:55:53.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=6,osdmap=94} crc {osd_snap=2606541058,osdmap=586596074}) 2024-09-16T10:55:53.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=3746196797,osdmap=3588588336}) 2024-09-16T10:55:53.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=6,osdmap=94} crc {osd_snap=2606541058,osdmap=586596074}) 2024-09-16T10:55:53.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=3746196797,osdmap=2135018811}) 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3330718952}) 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1694027922}) 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3330718952}) 2024-09-16T10:55:53.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.199+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1359468480}) 2024-09-16T10:55:53.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2710211854}) 2024-09-16T10:55:53.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2431540178}) 2024-09-16T10:55:53.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2710211854}) 2024-09-16T10:55:53.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=895588997}) 2024-09-16T10:55:53.208 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3813046451}) 2024-09-16T10:55:53.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3230033235}) 2024-09-16T10:55:53.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3813046451}) 2024-09-16T10:55:53.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.203+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3321891115}) 2024-09-16T10:55:53.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.215+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:55:53.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.215+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=10} crc {osdmap=328773797}) 2024-09-16T10:55:53.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:55:53.215+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=8} crc {osdmap=2860984091}) 2024-09-16T10:55:54.178 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:55:54.179 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 10 monclient: get_monmap_and_config 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 10 monclient: monmap: 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.299 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:54.299060+0000 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:54.299060+0000 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:54.300 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 1 Processor -- start 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 1 -- start start 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.295+0000 7f5410b35700 10 monclient: init 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T10:55:54.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-b con 0x7f540c0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 0x7f540c135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-c con 0x7f540c0a35a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f540c135f80 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0a3db0 con 0x7f540c135f80 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f540c0a4b70 con 0x7f540c0a35a0 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0a0850 con 0x7f540c0a2c80 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient(hunting): authenticate will time out at 1212.876757s 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44306/0 (socket says 172.21.15.5:44306) 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 -- 172.21.15.5:0/3928638754 learned_addr learned my addr 172.21.15.5:0/3928638754 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): get_auth_request con 0x7f540c0a2c80 auth_method 0 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540affd700 1 --2- 172.21.15.5:0/3928638754 >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 0x7f540c135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient(hunting): handle_auth_done global_id 54137 payload 274 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 -- 172.21.15.5:0/3928638754 >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 msgr2=0x7f540c135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:54.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 --2- 172.21.15.5:0/3928638754 >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 0x7f540c135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 -- 172.21.15.5:0/3928638754 >> v1:172.21.15.5:6789/0 conn(0x7f540c135f80 legacy=0x7f540c13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 monclient: found mon.noname-b 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 -- 172.21.15.5:0/3928638754 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0a4180 con 0x7f540c0a2c80 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540affd700 10 monclient: get_auth_request con 0x7f540c0a35a0 auth_method 0 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540affd700 1 --2- 172.21.15.5:0/3928638754 >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 0x7f540c135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540b7fe700 1 --2- 172.21.15.5:0/3928638754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e50 tx=0x7f540400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0f76eb94532b76e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 1 -- 172.21.15.5:0/3928638754 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400e070 con 0x7f540c0a2c80 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:54.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: dump: 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 5 monclient: authenticate success, global_id 54137 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 1 -- 172.21.15.5:0/3928638754 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400b9c0 con 0x7f540c0a2c80 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 1 -- 172.21.15.5:0/3928638754 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400bd70 con 0x7f540c0a2c80 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 4 set_mon_vals no callback set 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f540a7fc700 10 monclient: dump: 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.308 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.309 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 10 monclient: shutdown 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- 172.21.15.5:0/3928638754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 msgr2=0x7f540c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 --2- 172.21.15.5:0/3928638754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5404009e50 tx=0x7f540400b330 comp rx=0 tx=0).stop 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- 172.21.15.5:0/3928638754 shutdown_connections 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 --2- 172.21.15.5:0/3928638754 >> v2:172.21.15.143:3300/0 conn(0x7f540c0a35a0 0x7f540c135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 --2- 172.21.15.5:0/3928638754 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c0a2c80 0x7f540c0a3060 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- 172.21.15.5:0/3928638754 >> 172.21.15.5:0/3928638754 conn(0x7f540c0a4d80 msgr2=0x7f540c0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- 172.21.15.5:0/3928638754 shutdown_connections 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.299+0000 7f5410b35700 1 -- 172.21.15.5:0/3928638754 wait complete. 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: monmap: 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 librados: starting msgr at 2024-09-16T10:55:54.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 librados: starting objecter 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 Processor -- start 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- start start 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 librados: setting wanted keys 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 librados: calling monclient init 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: init 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 0x7f540c1ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-a con 0x7f540c0a2c80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 0x7f540c134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-b con 0x7f540c0a35a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 0x7f540c135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): picked mon.noname-c con 0x7f540c135f80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c1ca7b0 con 0x7f540c0a2c80 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540c126000 con 0x7f540c0a35a0 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540c0a7520 con 0x7f540c135f80 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient(hunting): authenticate will time out at 1212.881382s 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 0x7f540c1ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 0x7f540c135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 0x7f540c1ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59896/0 (socket says 172.21.15.5:59896) 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 1 -- 172.21.15.5:0/3214415320 learned_addr learned my addr 172.21.15.5:0/3214415320 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): get_auth_request con 0x7f540c0a2c80 auth_method 0 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): get_auth_request con 0x7f540c135f80 auth_method 0 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 0x7f540c134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 10 monclient(hunting): get_auth_request con 0x7f540c0a35a0 auth_method 0 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient(hunting): handle_auth_done global_id 54140 payload 824 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 -- 172.21.15.5:0/3214415320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 msgr2=0x7f540c134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 0x7f540c134bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 -- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 msgr2=0x7f540c1ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 0x7f540c1ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 monclient: found mon.noname-c 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 -- 172.21.15.5:0/3214415320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404009b30 con 0x7f540c135f80 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540affd700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 0x7f540c134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:55:54.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f540bfff700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 0x7f540c135100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f53fc001eb0 tx=0x7f53fc00c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 client.?.objecter ms_handle_connect 0x7f540c135f80 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0052d0 con 0x7f540c135f80 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: dump: 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.323 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.324 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.325 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 5 monclient: authenticate success, global_id 54140 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _renew_subs 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c09f4e0 con 0x7f540c135f80 2024-09-16T10:55:54.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc00ce90 con 0x7f540c135f80 2024-09-16T10:55:54.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0036a0 con 0x7f540c135f80 2024-09-16T10:55:54.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:54.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 client.54140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f53f3fff700 10 monclient: dump: 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.328 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _renew_subs 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c135940 con 0x7f540c135f80 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5410b35700 1 librados: init done 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.303+0000 7f5409ffb700 4 set_mon_vals no callback set 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f5410b35700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f5410b35700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f5410b35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f53fc003800 con 0x7f540c135f80 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540c126340 con 0x7f540c135f80 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 1 --2- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 0x7f53f8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f53fc047320 con 0x7f540c135f80 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 10 client.54140.objecter ms_dispatch 0x7f540c000fd0 osd_map(274..274 src has 1..274) v4 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 3 client.54140.objecter handle_osd_map got epochs [274,274] > 0 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 3 client.54140.objecter handle_osd_map decoding full epoch 274 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f540b7fe700 1 --2- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 0x7f53f8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f540b7fe700 10 monclient: get_auth_request con 0x7f53f8036f60 auth_method 0 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 20 client.54140.objecter dump_active .. 0 homeless 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f540b7fe700 1 --2- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 0x7f53f8039420 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f5404009440 tx=0x7f5404009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.307+0000 7f53f3fff700 10 client.54140.objecter ms_handle_connect 0x7f53f8036f60 2024-09-16T10:55:54.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.315+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f540c126340 con 0x7f540c135f80 2024-09-16T10:55:54.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.315+0000 7f53f3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.315+0000 7f53f3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.547+0000 7f5410b35700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.547+0000 7f5410b35700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.547+0000 7f5410b35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.547+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f540c126340 con 0x7f540c135f80 2024-09-16T10:55:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f53f3fff700 1 -- 172.21.15.5:0/3214415320 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f540c126340 con 0x7f540c135f80 2024-09-16T10:55:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f53f3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f53f3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:54.554 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 10 librados: watch_flush enter 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 10 librados: watch_flush exit 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 20 client.54140.objecter shutdown clearing up homeless session... 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 10 client.54140.objecter successfully canceled tick 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 msgr2=0x7f53f8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 0x7f53f8039420 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f5404009440 tx=0x7f5404009000 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 10 monclient: shutdown 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 msgr2=0x7f540c135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 0x7f540c135100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f53fc001eb0 tx=0x7f53fc00c6e0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 shutdown_connections 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f540c135f80 0x7f540c135100 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f540c0a35a0 0x7f540c134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f53f8036f60 0x7f53f8039420 unknown :-1 s=CLOSED pgs=3781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 --2- 172.21.15.5:0/3214415320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f540c0a2c80 0x7f540c1ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 >> 172.21.15.5:0/3214415320 conn(0x7f540c0a4d80 msgr2=0x7f540c0a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 shutdown_connections 2024-09-16T10:55:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 -- 172.21.15.5:0/3214415320 wait complete. 2024-09-16T10:55:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.551+0000 7f5410b35700 1 librados: shutdown 2024-09-16T10:55:54.568 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:55:54.568 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:55:54.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:55:54.568 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:55:54.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: get_monmap_and_config 2024-09-16T10:55:54.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: monmap: 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:54.685181+0000 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:54.685181+0000 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.686 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:54.687 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:54.687 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 Processor -- start 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- start start 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: init 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 0x7fd01c09e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-b con 0x7fd01c09aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd01c071870 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd01c1376e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c09caa0 con 0x7fd01c071870 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01c09e3a0 con 0x7fd01c1376e0 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c09ae30 con 0x7fd01c09aa50 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient(hunting): authenticate will time out at 1213.261554s 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd013fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd01c071870 0x7fd01c09ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36414/0 (socket says 172.21.15.5:36414) 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 0x7fd01c09e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd013fff700 1 -- 172.21.15.5:0/2320954864 learned_addr learned my addr 172.21.15.5:0/2320954864 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): get_auth_request con 0x7fd01c09aa50 auth_method 0 2024-09-16T10:55:54.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient(hunting): handle_auth_done global_id 54143 payload 274 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 1 -- 172.21.15.5:0/2320954864 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050262657 0 0) 0x7fd00c001110 con 0x7fd01c071870 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 -- 172.21.15.5:0/2320954864 >> v1:172.21.15.143:6789/0 conn(0x7fd01c1376e0 legacy=0x7fd01c139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 -- 172.21.15.5:0/2320954864 >> v1:172.21.15.5:6789/0 conn(0x7fd01c071870 legacy=0x7fd01c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 monclient: found mon.noname-b 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 -- 172.21.15.5:0/2320954864 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c09b5f0 con 0x7fd01c09aa50 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0208d4700 1 --2- 172.21.15.5:0/2320954864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 0x7fd01c09e8e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd01800a9a0 tx=0x7fd01800acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb713bc320542511 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 1 -- 172.21.15.5:0/2320954864 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018004230 con 0x7fd01c09aa50 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: dump: 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 5 monclient: authenticate success, global_id 54143 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 1 -- 172.21.15.5:0/2320954864 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd018004390 con 0x7fd01c09aa50 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 4 set_mon_vals no callback set 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 1 -- 172.21.15.5:0/2320954864 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01800c8f0 con 0x7fd01c09aa50 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0137fe700 10 monclient: dump: 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 10 monclient: shutdown 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- 172.21.15.5:0/2320954864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 msgr2=0x7fd01c09e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 --2- 172.21.15.5:0/2320954864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 0x7fd01c09e8e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd01800a9a0 tx=0x7fd01800acb0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- 172.21.15.5:0/2320954864 shutdown_connections 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 --2- 172.21.15.5:0/2320954864 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c09aa50 0x7fd01c09e8e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.683+0000 7fd0218d6700 1 -- 172.21.15.5:0/2320954864 >> 172.21.15.5:0/2320954864 conn(0x7fd01c0a45d0 msgr2=0x7fd01c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- 172.21.15.5:0/2320954864 shutdown_connections 2024-09-16T10:55:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- 172.21.15.5:0/2320954864 wait complete. 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient: monmap: 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:54.704 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 librados: starting msgr at 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 librados: starting objecter 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 Processor -- start 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- start start 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 librados: setting wanted keys 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 librados: calling monclient init 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient: init 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 0x7fd01c11ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-a con 0x7fd01c071870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd01c09aa50 0x7fd01c11f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-b con 0x7fd01c09aa50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c1376e0 0x7fd01c11f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): picked mon.noname-c con 0x7fd01c1376e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c1221a0 con 0x7fd01c071870 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c070530 con 0x7fd01c09aa50 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c136830 con 0x7fd01c1376e0 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0218d6700 10 monclient(hunting): authenticate will time out at 1213.266487s 2024-09-16T10:55:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 0x7fd01c11ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 0x7fd01c11ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59902/0 (socket says 172.21.15.5:59902) 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 -- 172.21.15.5:0/3271932440 learned_addr learned my addr 172.21.15.5:0/3271932440 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): get_auth_request con 0x7fd01c071870 auth_method 0 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c1376e0 0x7fd01c11f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd01c09aa50 0x7fd01c11f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): get_auth_request con 0x7fd01c09aa50 auth_method 0 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): get_auth_request con 0x7fd01c1376e0 auth_method 0 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0210d5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient(hunting): handle_auth_done global_id 28746 payload 824 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c1376e0 msgr2=0x7fd01c11f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd01c1376e0 0x7fd01c11f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd01c09aa50 msgr2=0x7fd01c11f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd01c09aa50 0x7fd01c11f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 monclient: found mon.noname-a 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01800a650 con 0x7fd01c071870 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0208d4700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 0x7fd01c11ee50 secure :-1 s=READY pgs=6844 cs=0 l=1 rev1=1 crypto rx=0x7fd01c075050 tx=0x7fd0180047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0117fa700 10 client.?.objecter ms_handle_connect 0x7fd01c071870 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.687+0000 7fd0117fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01801e070 con 0x7fd01c071870 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: dump: 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 5 monclient: authenticate success, global_id 28746 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0180038d0 con 0x7fd01c071870 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 monclient: _renew_subs 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c0a34a0 con 0x7fd01c071870 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018003ca0 con 0x7fd01c071870 2024-09-16T10:55:54.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd012ffd700 4 set_mon_vals no callback set 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: dump: 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.712 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 client.28746.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 monclient: _renew_subs 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c122a40 con 0x7fd01c071870 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 1 librados: init done 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 0x7fd01c11a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c067380 con 0x7fd01c11a530 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 0x7fd01c11a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd0180183f0 con 0x7fd01c071870 2024-09-16T10:55:54.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient: get_auth_request con 0x7fd01c11a530 auth_method 0 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 --2- 172.21.15.5:0/3271932440 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd008036f60 0x7fd008039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fd0180494d0 con 0x7fd01c071870 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 client.28746.objecter ms_dispatch 0x7fd01c000fd0 osd_map(274..274 src has 1..274) v4 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 3 client.28746.objecter handle_osd_map got epochs [274,274] > 0 2024-09-16T10:55:54.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 3 client.28746.objecter handle_osd_map decoding full epoch 274 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 10 monclient(hunting): handle_auth_done global_id 28749 payload 824 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd01c09d580 con 0x7fd01c11a530 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 20 client.28746.objecter dump_active .. 0 homeless 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0210d5700 1 --2- 172.21.15.5:0/3271932440 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd008036f60 0x7fd008039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0210d5700 10 monclient: get_auth_request con 0x7fd008036f60 auth_method 0 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 client.28746.objecter ms_handle_connect 0x7fd01c11a530 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 client.28746.objecter resend_mon_ops 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd013fff700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 0x7fd01c11a910 secure :-1 s=READY pgs=6845 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000f60 tx=0x7fd00c001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c017070 con 0x7fd01c11a530 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 monclient: dump: 2024-09-16T10:55:54.715 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0210d5700 1 --2- 172.21.15.5:0/3271932440 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd008036f60 0x7fd008039420 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09bb30 tx=0x7fd014009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 10 client.28746.objecter ms_handle_connect 0x7fd008036f60 2024-09-16T10:55:54.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.691+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd00c012560 con 0x7fd01c11a530 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 msgr2=0x7fd01c11a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 0x7fd01c11a910 secure :-1 s=READY pgs=6845 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000f60 tx=0x7fd00c001860 comp rx=0 tx=0).stop 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 msgr2=0x7fd01c11a910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.695+0000 7fd0117fa700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11a530 0x7fd01c11a910 secure :-1 s=CLOSED pgs=6845 cs=0 l=1 rev1=1 crypto rx=0x7fd00c000f60 tx=0x7fd00c001860 comp rx=0 tx=0).stop 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0218d6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0218d6700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 0x7fd01c1d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0218d6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c09c250 con 0x7fd01c11ae50 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 0x7fd01c1d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient: get_auth_request con 0x7fd01c11ae50 auth_method 0 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 10 monclient(hunting): handle_auth_done global_id 28752 payload 824 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 1 -- 172.21.15.5:0/3271932440 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd01c0a0680 con 0x7fd01c11ae50 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0117fa700 10 client.28746.objecter ms_handle_connect 0x7fd01c11ae50 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd0117fa700 10 client.28746.objecter resend_mon_ops 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.703+0000 7fd013fff700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 0x7fd01c1d08c0 secure :-1 s=READY pgs=6846 cs=0 l=1 rev1=1 crypto rx=0x7fd00c016000 tx=0x7fd00c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c023070 con 0x7fd01c11ae50 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 10 monclient: dump: 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7fd00c012d70 con 0x7fd01c11ae50 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 msgr2=0x7fd01c1d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 0x7fd01c1d08c0 secure :-1 s=READY pgs=6846 cs=0 l=1 rev1=1 crypto rx=0x7fd00c016000 tx=0x7fd00c000ed0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 msgr2=0x7fd01c1d08c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0117fa700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c11ae50 0x7fd01c1d08c0 secure :-1 s=CLOSED pgs=6846 cs=0 l=1 rev1=1 crypto rx=0x7fd00c016000 tx=0x7fd00c000ed0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.722 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:54.723 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.724 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.725 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:54.726 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.727 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.728 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.729 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.730 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.731 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:54.732 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:54.733 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:55:54.734 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:54.735 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 10 librados: watch_flush enter 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 10 librados: watch_flush exit 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 20 client.28746.objecter shutdown clearing up homeless session... 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 10 client.28746.objecter successfully canceled tick 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd008036f60 msgr2=0x7fd008039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 --2- 172.21.15.5:0/3271932440 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd008036f60 0x7fd008039420 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7fd01c09bb30 tx=0x7fd014009040 comp rx=0 tx=0).stop 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 10 monclient: shutdown 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 msgr2=0x7fd01c11ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 --2- 172.21.15.5:0/3271932440 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd01c071870 0x7fd01c11ee50 secure :-1 s=READY pgs=6844 cs=0 l=1 rev1=1 crypto rx=0x7fd01c075050 tx=0x7fd0180047e0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0210d5700 1 -- 172.21.15.5:0/3271932440 reap_dead start 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0210d5700 1 -- 172.21.15.5:0/3271932440 reap_dead start 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 shutdown_connections 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 >> 172.21.15.5:0/3271932440 conn(0x7fd01c0a45d0 msgr2=0x7fd01c09fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 shutdown_connections 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 -- 172.21.15.5:0/3271932440 wait complete. 2024-09-16T10:55:54.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.707+0000 7fd0218d6700 1 librados: shutdown 2024-09-16T10:55:54.738 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:54.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:55:54.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: get_monmap_and_config 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: monmap: 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:54.853721+0000 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:54.853721+0000 2024-09-16T10:55:54.854 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:54.855 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 Processor -- start 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- start start 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: init 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd5fc0a35a0 0x7fd5fc135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): picked mon.noname-a con 0x7fd5fc0a35a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5fc135f80 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc0a3db0 con 0x7fd5fc135f80 2024-09-16T10:55:54.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc0a4b70 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0a0850 con 0x7fd5fc0a2c80 2024-09-16T10:55:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient(hunting): authenticate will time out at 1213.430021s 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd5fc0a35a0 0x7fd5fc135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601e65700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd5fc135f80 0x7fd5fc13cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36416/0 (socket says 172.21.15.5:36416) 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601e65700 1 -- 172.21.15.5:0/2011776624 learned_addr learned my addr 172.21.15.5:0/2011776624 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): get_auth_request con 0x7fd5fc0a35a0 auth_method 0 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 10 monclient(hunting): get_auth_request con 0x7fd5fc0a2c80 auth_method 0 2024-09-16T10:55:54.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient(hunting): handle_auth_done global_id 28755 payload 274 2024-09-16T10:55:54.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 -- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 msgr2=0x7fd5fc0a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc0a3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 1 -- 172.21.15.5:0/2011776624 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776119111 0 0) 0x7fd5fc0a3db0 con 0x7fd5fc135f80 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 -- 172.21.15.5:0/2011776624 >> v1:172.21.15.5:6789/0 conn(0x7fd5fc135f80 legacy=0x7fd5fc13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 monclient: found mon.noname-a 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 -- 172.21.15.5:0/2011776624 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc0a4180 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd601664700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc0a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd600e63700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc135a40 secure :-1 s=READY pgs=6848 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009850 tx=0x7fd5ec009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d1a99616647b1f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 1 -- 172.21.15.5:0/2011776624 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec011830 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: dump: 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.874 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 5 monclient: authenticate success, global_id 28755 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 1 -- 172.21.15.5:0/2011776624 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec011990 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 4 set_mon_vals no callback set 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 1 -- 172.21.15.5:0/2011776624 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec011d20 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd5f3fff700 10 monclient: dump: 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 10 monclient: shutdown 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc0a35a0 msgr2=0x7fd5fc135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc135a40 secure :-1 s=READY pgs=6848 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009850 tx=0x7fd5ec009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- 172.21.15.5:0/2011776624 shutdown_connections 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc135a40 unknown :-1 s=CLOSED pgs=6848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 --2- 172.21.15.5:0/2011776624 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc0a3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.851+0000 7fd602666700 1 -- 172.21.15.5:0/2011776624 >> 172.21.15.5:0/2011776624 conn(0x7fd5fc0a4d80 msgr2=0x7fd5fc0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- 172.21.15.5:0/2011776624 shutdown_connections 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- 172.21.15.5:0/2011776624 wait complete. 2024-09-16T10:55:54.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient: build_initial_monmap 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient: monmap: 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 librados: starting msgr at 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 librados: starting objecter 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 Processor -- start 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- start start 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 librados: setting wanted keys 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 librados: calling monclient init 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient: init 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc1caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): picked mon.noname-c con 0x7fd5fc0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc1caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc0a35a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc135f80 0x7fd5fc1220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): picked mon.noname-a con 0x7fd5fc135f80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc124340 con 0x7fd5fc135f80 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc125eb0 con 0x7fd5fc0a35a0 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc124940 con 0x7fd5fc0a2c80 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd602666700 10 monclient(hunting): authenticate will time out at 1213.434990s 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc1caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc135f80 0x7fd5fc1220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc1caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44356/0 (socket says 172.21.15.5:44356) 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 1 -- 172.21.15.5:0/3496799772 learned_addr learned my addr 172.21.15.5:0/3496799772 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): get_auth_request con 0x7fd5fc0a2c80 auth_method 0 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): get_auth_request con 0x7fd5fc135f80 auth_method 0 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc1caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): get_auth_request con 0x7fd5fc0a35a0 auth_method 0 2024-09-16T10:55:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd600e63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient(hunting): handle_auth_done global_id 28761 payload 824 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 msgr2=0x7fd5fc1caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc1caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5fc0a35a0 msgr2=0x7fd5fc1caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5fc0a35a0 0x7fd5fc1caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient: _finish_hunting 0 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 monclient: found mon.noname-a 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec0094b0 con 0x7fd5fc135f80 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601664700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc0a2c80 0x7fd5fc1caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd5f1ffb700 10 client.?.objecter ms_handle_connect 0x7fd5fc135f80 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd5f1ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.855+0000 7fd601e65700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc135f80 0x7fd5fc1220d0 secure :-1 s=READY pgs=6849 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400cbe0 tx=0x7fd5f400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400aaa0 con 0x7fd5fc135f80 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: _finish_auth 0 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f400ac00 con 0x7fd5fc135f80 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40070d0 con 0x7fd5fc135f80 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:54.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 5 monclient: authenticate success, global_id 28761 2024-09-16T10:55:54.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 monclient: _renew_subs 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f37fe700 4 set_mon_vals no callback set 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc0a7310 con 0x7fd5fc135f80 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 client.28761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 monclient: _renew_subs 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc123d70 con 0x7fd5fc135f80 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 1 librados: init done 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 0x7fd5fc1229f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc126260 con 0x7fd5fc122610 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd5f4007230 con 0x7fd5fc135f80 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 0x7fd5fc1229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient: get_auth_request con 0x7fd5fc122610 auth_method 0 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 --2- 172.21.15.5:0/3496799772 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd5e0036f90 0x7fd5e0039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7fd5f4047970 con 0x7fd5fc135f80 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 client.28761.objecter ms_dispatch 0x7fd5fc000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 3 client.28761.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 3 client.28761.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 20 client.28761.objecter dump_active .. 0 homeless 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 10 monclient(hunting): handle_auth_done global_id 54155 payload 824 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5fc09fa10 con 0x7fd5fc122610 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd600e63700 1 --2- 172.21.15.5:0/3496799772 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd5e0036f90 0x7fd5e0039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd600e63700 10 monclient: get_auth_request con 0x7fd5e0036f90 auth_method 0 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 client.28761.objecter ms_handle_connect 0x7fd5fc122610 2024-09-16T10:55:54.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 client.28761.objecter resend_mon_ops 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd601664700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 0x7fd5fc1229f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8003c60 tx=0x7fd5f80104d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8010c60 con 0x7fd5fc122610 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.892 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 client.28761.objecter ms_handle_connect 0x7fd5e0036f90 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd600e63700 1 --2- 172.21.15.5:0/3496799772 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd5e0036f90 0x7fd5e0039450 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd5f8010dc0 con 0x7fd5fc122610 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 msgr2=0x7fd5fc1229f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 0x7fd5fc1229f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8003c60 tx=0x7fd5f80104d0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 msgr2=0x7fd5fc1229f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.859+0000 7fd5f1ffb700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122610 0x7fd5fc1229f0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8003c60 tx=0x7fd5f80104d0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.867+0000 7fd602666700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.867+0000 7fd602666700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 0x7fd5fc11e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.867+0000 7fd602666700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.867+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc067380 con 0x7fd5fc122f30 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.867+0000 7fd601664700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 0x7fd5fc11e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient: get_auth_request con 0x7fd5fc122f30 auth_method 0 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 10 monclient(hunting): handle_auth_done global_id 54158 payload 824 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 1 -- 172.21.15.5:0/3496799772 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd5fc0a4400 con 0x7fd5fc122f30 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601664700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 0x7fd5fc11e8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8003c60 tx=0x7fd5f8004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 client.28761.objecter ms_handle_connect 0x7fd5fc122f30 2024-09-16T10:55:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 client.28761.objecter resend_mon_ops 2024-09-16T10:55:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8015be0 con 0x7fd5fc122f30 2024-09-16T10:55:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-16T10:55:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7fd5f8015d40 con 0x7fd5fc122f30 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 msgr2=0x7fd5fc11e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 0x7fd5fc11e8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8003c60 tx=0x7fd5f8004770 comp rx=0 tx=0).stop 2024-09-16T10:55:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 msgr2=0x7fd5fc11e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd5f1ffb700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd5fc122f30 0x7fd5fc11e8d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:54.903 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:55:54.904 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.906 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.907 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:54.909 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:54.910 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:54.910 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:54.910 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:54.910 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:54.911 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:54.912 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:54.912 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:54.912 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:54.912 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:54.923 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 10 librados: watch_flush enter 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 10 librados: watch_flush exit 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 20 client.28761.objecter shutdown clearing up homeless session... 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 10 client.28761.objecter successfully canceled tick 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd5e0036f90 msgr2=0x7fd5e0039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 --2- 172.21.15.5:0/3496799772 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd5e0036f90 0x7fd5e0039450 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec000c00 tx=0x7fd5ec008030 comp rx=0 tx=0).stop 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 10 monclient: shutdown 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc135f80 msgr2=0x7fd5fc1220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:54.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 --2- 172.21.15.5:0/3496799772 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5fc135f80 0x7fd5fc1220d0 secure :-1 s=READY pgs=6849 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400cbe0 tx=0x7fd5f400cef0 comp rx=0 tx=0).stop 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601e65700 1 -- 172.21.15.5:0/3496799772 reap_dead start 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd601e65700 1 -- 172.21.15.5:0/3496799772 reap_dead start 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 shutdown_connections 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 >> 172.21.15.5:0/3496799772 conn(0x7fd5fc0a4d80 msgr2=0x7fd5fc0a6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 shutdown_connections 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 -- 172.21.15.5:0/3496799772 wait complete. 2024-09-16T10:55:54.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:54.871+0000 7fd602666700 1 librados: shutdown 2024-09-16T10:55:54.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:54.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:55:54.927 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:55:55.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: get_monmap_and_config 2024-09-16T10:55:55.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: monmap: 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:55.049306+0000 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:55.049306+0000 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:55.051 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 Processor -- start 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 -- start start 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: init 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-b con 0x7f710c0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f710c0a35a0 0x7f710c135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-c con 0x7f710c0a35a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f710c135f80 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0a3db0 con 0x7f710c135f80 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710c0a4b70 con 0x7f710c0a35a0 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0a0850 con 0x7f710c0a2c80 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f71108f7700 10 monclient(hunting): authenticate will time out at 1213.626541s 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710bfff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f710c135f80 0x7f710c13cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36430/0 (socket says 172.21.15.5:36430) 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710bfff700 1 -- 172.21.15.5:0/2116482435 learned_addr learned my addr 172.21.15.5:0/2116482435 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 1 --2- 172.21.15.5:0/2116482435 >> v2:172.21.15.143:3300/0 conn(0x7f710c0a35a0 0x7f710c135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): get_auth_request con 0x7f710c0a2c80 auth_method 0 2024-09-16T10:55:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): get_auth_request con 0x7f710c0a35a0 auth_method 0 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 1 -- 172.21.15.5:0/2116482435 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295250166 0 0) 0x7f710c0a3db0 con 0x7f710c135f80 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 10 monclient(hunting): _negotiate 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 10 monclient(hunting): my global_id is 28764 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710a7fc700 1 -- 172.21.15.5:0/2116482435 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f80035d0 con 0x7f710c135f80 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient(hunting): handle_auth_done global_id 54161 payload 274 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 -- 172.21.15.5:0/2116482435 >> v2:172.21.15.143:3300/0 conn(0x7f710c0a35a0 msgr2=0x7f710c135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 --2- 172.21.15.5:0/2116482435 >> v2:172.21.15.143:3300/0 conn(0x7f710c0a35a0 0x7f710c135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 -- 172.21.15.5:0/2116482435 >> v1:172.21.15.5:6789/0 conn(0x7f710c135f80 legacy=0x7f710c13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 monclient: found mon.noname-b 2024-09-16T10:55:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 -- 172.21.15.5:0/2116482435 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c0a4180 con 0x7f710c0a2c80 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.047+0000 7f710b7fe700 1 --2- 172.21.15.5:0/2116482435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c0a3060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009850 tx=0x7f70fc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a5228690688a2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 1 -- 172.21.15.5:0/2116482435 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc011830 con 0x7f710c0a2c80 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: dump: 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 5 monclient: authenticate success, global_id 54161 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 1 -- 172.21.15.5:0/2116482435 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc011990 con 0x7f710c0a2c80 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 1 -- 172.21.15.5:0/2116482435 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc011d40 con 0x7f710c0a2c80 2024-09-16T10:55:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 4 set_mon_vals no callback set 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710a7fc700 10 monclient: dump: 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: shutdown 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- 172.21.15.5:0/2116482435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 msgr2=0x7f710c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- 172.21.15.5:0/2116482435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c0a3060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009850 tx=0x7f70fc009b60 comp rx=0 tx=0).stop 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- 172.21.15.5:0/2116482435 shutdown_connections 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- 172.21.15.5:0/2116482435 >> v2:172.21.15.143:3300/0 conn(0x7f710c0a35a0 0x7f710c135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- 172.21.15.5:0/2116482435 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c0a3060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- 172.21.15.5:0/2116482435 >> 172.21.15.5:0/2116482435 conn(0x7f710c0a4d80 msgr2=0x7f710c0a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- 172.21.15.5:0/2116482435 shutdown_connections 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- 172.21.15.5:0/2116482435 wait complete. 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: monmap: 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 librados: starting msgr at 2024-09-16T10:55:55.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 librados: starting objecter 2024-09-16T10:55:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 Processor -- start 2024-09-16T10:55:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- start start 2024-09-16T10:55:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 librados: setting wanted keys 2024-09-16T10:55:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 librados: calling monclient init 2024-09-16T10:55:55.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: init 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c1ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-c con 0x7f710c0a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710c0a35a0 0x7f710c134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-a con 0x7f710c0a35a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c135f80 0x7f710c135280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): picked mon.noname-b con 0x7f710c135f80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c1ca760 con 0x7f710c0a35a0 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0a7090 con 0x7f710c135f80 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c1251d0 con 0x7f710c0a2c80 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f71108f7700 10 monclient(hunting): authenticate will time out at 1213.631767s 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c1ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710affd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710c0a35a0 0x7f710c134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c1ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44388/0 (socket says 172.21.15.5:44388) 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.051+0000 7f710b7fe700 1 -- 172.21.15.5:0/4094488805 learned_addr learned my addr 172.21.15.5:0/4094488805 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): get_auth_request con 0x7f710c0a2c80 auth_method 0 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): get_auth_request con 0x7f710c0a35a0 auth_method 0 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c135f80 0x7f710c135280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): get_auth_request con 0x7f710c135f80 auth_method 0 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient(hunting): handle_auth_done global_id 54164 payload 824 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c135f80 msgr2=0x7f710c135280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c135f80 0x7f710c135280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710c0a35a0 msgr2=0x7f710c134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710c0a35a0 0x7f710c134d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 monclient: found mon.noname-c 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0094b0 con 0x7f710c0a2c80 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710c0a35a0 0x7f710c134d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710b7fe700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c1ca220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f710c072830 tx=0x7f70fc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 client.?.objecter ms_handle_connect 0x7f710c0a2c80 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc004510 con 0x7f710c0a2c80 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: dump: 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.077 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc0036c0 con 0x7f710c0a2c80 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 5 monclient: authenticate success, global_id 54164 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 monclient: _renew_subs 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c09f4e0 con 0x7f710c0a2c80 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc003a70 con 0x7f710c0a2c80 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 monclient: dump: 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71097fa700 4 set_mon_vals no callback set 2024-09-16T10:55:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 client.54164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 monclient: _renew_subs 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c126740 con 0x7f710c0a2c80 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 1 librados: init done 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f70fc021070 con 0x7f710c0a2c80 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 0x7f710c122b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c1268d0 con 0x7f710c122790 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 --2- 172.21.15.5:0/4094488805 >> v2:172.21.15.143:6812/1098205034 conn(0x7f70ec036f60 0x7f70ec039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f70fc049490 con 0x7f710c0a2c80 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 10 client.54164.objecter ms_dispatch 0x7f710c000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:55:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 3 client.54164.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 3 client.54164.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f7103fff700 20 client.54164.objecter dump_active .. 0 homeless 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 0x7f710c122b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient: get_auth_request con 0x7f710c122790 auth_method 0 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 1 --2- 172.21.15.5:0/4094488805 >> v2:172.21.15.143:6812/1098205034 conn(0x7f70ec036f60 0x7f70ec039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710bfff700 10 monclient: get_auth_request con 0x7f70ec036f60 auth_method 0 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 10 monclient(hunting): handle_auth_done global_id 44407 payload 824 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.055+0000 7f710affd700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f710c125f30 con 0x7f710c122790 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 client.54164.objecter ms_handle_connect 0x7f70ec036f60 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f710bfff700 1 --2- 172.21.15.5:0/4094488805 >> v2:172.21.15.143:6812/1098205034 conn(0x7f70ec036f60 0x7f70ec039420 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7f7104001560 tx=0x7f7104001a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 client.54164.objecter ms_handle_connect 0x7f710c122790 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 client.54164.objecter resend_mon_ops 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f710affd700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 0x7f710c122b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f400f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f400fc90 con 0x7f710c122790 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 monclient: dump: 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.082 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f70f400fdf0 con 0x7f710c122790 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 msgr2=0x7f710c122b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 0x7f710c122b70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f400f520 comp rx=0 tx=0).stop 2024-09-16T10:55:55.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 msgr2=0x7f710c122b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:55.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.059+0000 7f7103fff700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c122790 0x7f710c122b70 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f400f520 comp rx=0 tx=0).stop 2024-09-16T10:55:55.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 0x7f710c11e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c067380 con 0x7f710c1230b0 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 0x7f710c11e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient: get_auth_request con 0x7f710c1230b0 auth_method 0 2024-09-16T10:55:55.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 10 monclient(hunting): handle_auth_done global_id 44410 payload 824 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 1 -- 172.21.15.5:0/4094488805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f710c09fa10 con 0x7f710c1230b0 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710affd700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 0x7f710c11e8d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 client.54164.objecter ms_handle_connect 0x7f710c1230b0 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 client.54164.objecter resend_mon_ops 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4014640 con 0x7f710c1230b0 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 monclient: dump: 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.088 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f70f40273a0 con 0x7f710c1230b0 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 msgr2=0x7f710c11e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 0x7f710c11e8d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f4004770 comp rx=0 tx=0).stop 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 msgr2=0x7f710c11e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:55.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f7103fff700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f710c1230b0 0x7f710c11e8d0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f710c123fe0 tx=0x7f70f4004770 comp rx=0 tx=0).stop 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 30, 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.090 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:55.091 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:55:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:55:55.094 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:55.095 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:55:55.096 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:55:55.097 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:55.098 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:55:55.099 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:55:55.100 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.101 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:55.102 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:55:55.103 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:55:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:55:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:55:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:55:55.104 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:55:55.104 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 librados: watch_flush enter 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 librados: watch_flush exit 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 20 client.54164.objecter shutdown clearing up homeless session... 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 client.54164.objecter successfully canceled tick 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 >> v2:172.21.15.143:6812/1098205034 conn(0x7f70ec036f60 msgr2=0x7f70ec039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 --2- 172.21.15.5:0/4094488805 >> v2:172.21.15.143:6812/1098205034 conn(0x7f70ec036f60 0x7f70ec039420 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7f7104001560 tx=0x7f7104001a10 comp rx=0 tx=0).stop 2024-09-16T10:55:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 10 monclient: shutdown 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 msgr2=0x7f710c1ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 --2- 172.21.15.5:0/4094488805 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f710c0a2c80 0x7f710c1ca220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f710c072830 tx=0x7f70fc004770 comp rx=0 tx=0).stop 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710bfff700 1 -- 172.21.15.5:0/4094488805 reap_dead start 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f710bfff700 1 -- 172.21.15.5:0/4094488805 reap_dead start 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 shutdown_connections 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 >> 172.21.15.5:0/4094488805 conn(0x7f710c0a4d80 msgr2=0x7f710c0a6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 shutdown_connections 2024-09-16T10:55:55.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 -- 172.21.15.5:0/4094488805 wait complete. 2024-09-16T10:55:55.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.071+0000 7f71108f7700 1 librados: shutdown 2024-09-16T10:55:55.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:55:55.110 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T10:55:55.110 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:55:55.110 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T10:55:55.110 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T10:55:55.110 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T10:55:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: get_monmap_and_config 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: monmap: 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:55.234315+0000 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:55.234315+0000 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:55.236 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:55.236 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 Processor -- start 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 -- start start 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: init 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T10:55:55.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb25409b960 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2540a2700 0x7fb25413b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-c con 0x7fb2540a2700 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-b con 0x7fb25413b780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2540a4c90 con 0x7fb25409b960 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2540a6120 con 0x7fb2540a2700 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540a1740 con 0x7fb25413b780 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25c4b5700 10 monclient(hunting): authenticate will time out at 1213.811420s 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44390/0 (socket says 172.21.15.5:44390) 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 1 -- 172.21.15.5:0/1192961076 learned_addr learned my addr 172.21.15.5:0/1192961076 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): get_auth_request con 0x7fb25413b780 auth_method 0 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb259a50700 1 --2- 172.21.15.5:0/1192961076 >> v2:172.21.15.143:3300/0 conn(0x7fb2540a2700 0x7fb25413b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb259a50700 10 monclient(hunting): get_auth_request con 0x7fb2540a2700 auth_method 0 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb259a50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb259a50700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.231+0000 7fb259a50700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 10 monclient(hunting): handle_auth_done global_id 54167 payload 274 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 -- 172.21.15.5:0/1192961076 >> v2:172.21.15.143:3300/0 conn(0x7fb2540a2700 msgr2=0x7fb25413b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 --2- 172.21.15.5:0/1192961076 >> v2:172.21.15.143:3300/0 conn(0x7fb2540a2700 0x7fb25413b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 -- 172.21.15.5:0/1192961076 >> v1:172.21.15.143:6789/0 conn(0x7fb25409b960 legacy=0x7fb25409bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 monclient: found mon.noname-b 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 -- 172.21.15.5:0/1192961076 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2540a1d70 con 0x7fb25413b780 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25aa52700 1 --2- 172.21.15.5:0/1192961076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009e50 tx=0x7fb24c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90fdac793898263e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 1 -- 172.21.15.5:0/1192961076 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c00e070 con 0x7fb25413b780 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: dump: 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.243 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 1 -- 172.21.15.5:0/1192961076 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c00b910 con 0x7fb25413b780 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 5 monclient: authenticate success, global_id 54167 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:55.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 1 -- 172.21.15.5:0/1192961076 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c00bcc0 con 0x7fb25413b780 2024-09-16T10:55:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 4 set_mon_vals no callback set 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25924f700 10 monclient: dump: 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.246 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 10 monclient: shutdown 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1192961076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 msgr2=0x7fb25413db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1192961076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009e50 tx=0x7fb24c00b330 comp rx=0 tx=0).stop 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1192961076 shutdown_connections 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1192961076 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25413db70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1192961076 >> v2:172.21.15.143:3300/0 conn(0x7fb2540a2700 0x7fb25413b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1192961076 >> 172.21.15.5:0/1192961076 conn(0x7fb2540a5330 msgr2=0x7fb2540a5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1192961076 shutdown_connections 2024-09-16T10:55:55.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1192961076 wait complete. 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 10 monclient: monmap: 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:55.257 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 librados: starting msgr at 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.235+0000 7fb25c4b5700 1 librados: starting objecter 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 Processor -- start 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- start start 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 librados: setting wanted keys 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 librados: calling monclient init 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: init 2024-09-16T10:55:55.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25409b960 0x7fb2540987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-b con 0x7fb25409b960 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2540a2700 0x7fb254098d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-a con 0x7fb2540a2700 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25406dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): picked mon.noname-c con 0x7fb25413b780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540710f0 con 0x7fb2540a2700 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25413dda0 con 0x7fb25409b960 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25409fa20 con 0x7fb25413b780 2024-09-16T10:55:55.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): authenticate will time out at 1213.817279s 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25406dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2540a2700 0x7fb254098d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25406dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44404/0 (socket says 172.21.15.5:44404) 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 1 -- 172.21.15.5:0/1472965224 learned_addr learned my addr 172.21.15.5:0/1472965224 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): get_auth_request con 0x7fb25413b780 auth_method 0 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): get_auth_request con 0x7fb2540a2700 auth_method 0 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25409b960 0x7fb2540987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): get_auth_request con 0x7fb25409b960 auth_method 0 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient(hunting): handle_auth_done global_id 28770 payload 824 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 msgr2=0x7fb25406dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25406dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25409b960 msgr2=0x7fb2540987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25409b960 0x7fb2540987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 monclient: found mon.noname-a 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c009b30 con 0x7fb2540a2700 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25aa52700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb25413b780 0x7fb25406dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb259a50700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2540a2700 0x7fb254098d20 secure :-1 s=READY pgs=6851 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e900 tx=0x7fb25000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 client.?.objecter ms_handle_connect 0x7fb2540a2700 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250013680 con 0x7fb2540a2700 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: dump: 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.263 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 5 monclient: authenticate success, global_id 28770 2024-09-16T10:55:55.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2500137e0 con 0x7fb2540a2700 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _renew_subs 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2540a8130 con 0x7fb2540a2700 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250013b50 con 0x7fb2540a2700 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24bfff700 4 set_mon_vals no callback set 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb24affd700 10 monclient: dump: 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 client.28770.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _renew_subs 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25412df70 con 0x7fb2540a2700 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 librados: init done 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 0x7fb25406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.239+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540677c0 con 0x7fb25406dff0 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb250013cb0 con 0x7fb2540a2700 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 0x7fb25406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient: get_auth_request con 0x7fb25406dff0 auth_method 0 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 --2- 172.21.15.5:0/1472965224 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb238036f60 0x7fb238039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7fb250048330 con 0x7fb2540a2700 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 client.28770.objecter ms_dispatch 0x7fb254000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 3 client.28770.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:55:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 3 client.28770.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 10 monclient(hunting): handle_auth_done global_id 44422 payload 824 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 20 client.28770.objecter dump_active .. 0 homeless 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2540a84f0 con 0x7fb25406dff0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25aa52700 1 --2- 172.21.15.5:0/1472965224 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb238036f60 0x7fb238039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25aa52700 10 monclient: get_auth_request con 0x7fb238036f60 auth_method 0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 client.28770.objecter ms_handle_connect 0x7fb25406dff0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 client.28770.objecter resend_mon_ops 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25a251700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 0x7fb25406e3d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb24400f770 tx=0x7fb24400fb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2440052e0 con 0x7fb25406dff0 2024-09-16T10:55:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 monclient: dump: 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 client.28770.objecter ms_handle_connect 0x7fb238036f60 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb25aa52700 1 --2- 172.21.15.5:0/1472965224 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb238036f60 0x7fb238039420 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009440 tx=0x7fb24c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fb244005440 con 0x7fb25406dff0 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 msgr2=0x7fb25406e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 0x7fb25406e3d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb24400f770 tx=0x7fb24400fb30 comp rx=0 tx=0).stop 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 msgr2=0x7fb25406e3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:55.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.243+0000 7fb24affd700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406dff0 0x7fb25406e3d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25c4b5700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 0x7fb25409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25c4b5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540a14f0 con 0x7fb25406e910 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 0x7fb25409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient: get_auth_request con 0x7fb25406e910 auth_method 0 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 10 monclient(hunting): handle_auth_done global_id 44425 payload 824 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.255+0000 7fb25a251700 1 -- 172.21.15.5:0/1472965224 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fb254066920 con 0x7fb25406e910 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 client.28770.objecter ms_handle_connect 0x7fb25406e910 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25a251700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 0x7fb25409c6a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2440015a0 tx=0x7fb244001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 client.28770.objecter resend_mon_ops 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244023070 con 0x7fb25406e910 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 monclient: dump: 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (secure 0 0 0) 0x7fb244006880 con 0x7fb25406e910 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 msgr2=0x7fb25409c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 0x7fb25409c6a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2440015a0 tx=0x7fb244001120 comp rx=0 tx=0).stop 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 msgr2=0x7fb25409c6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:55:55.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb24affd700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb25406e910 0x7fb25409c6a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.277 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:55:55.277 INFO:teuthology.orchestra.run.smithi005.stdout: "ret": 0, 2024-09-16T10:55:55.277 INFO:teuthology.orchestra.run.smithi005.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T10:55:55.277 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:55:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 10 librados: watch_flush enter 2024-09-16T10:55:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 10 librados: watch_flush exit 2024-09-16T10:55:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 20 client.28770.objecter shutdown clearing up homeless session... 2024-09-16T10:55:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 10 client.28770.objecter successfully canceled tick 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb238036f60 msgr2=0x7fb238039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1472965224 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb238036f60 0x7fb238039420 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009440 tx=0x7fb24c009000 comp rx=0 tx=0).stop 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 10 monclient: shutdown 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2540a2700 msgr2=0x7fb254098d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 --2- 172.21.15.5:0/1472965224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2540a2700 0x7fb254098d20 secure :-1 s=READY pgs=6851 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e900 tx=0x7fb25000ec10 comp rx=0 tx=0).stop 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25aa52700 1 -- 172.21.15.5:0/1472965224 reap_dead start 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25aa52700 1 -- 172.21.15.5:0/1472965224 reap_dead start 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 shutdown_connections 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 >> 172.21.15.5:0/1472965224 conn(0x7fb2540a5330 msgr2=0x7fb254099a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 shutdown_connections 2024-09-16T10:55:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 -- 172.21.15.5:0/1472965224 wait complete. 2024-09-16T10:55:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.259+0000 7fb25c4b5700 1 librados: shutdown 2024-09-16T10:55:55.281 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:55:55.281 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T10:55:55.281 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:55:55.315 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T10:55:55.315 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:55:55.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: get_monmap_and_config 2024-09-16T10:55:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: monmap: 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:55.434066+0000 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:55.434066+0000 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:55.435 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 Processor -- start 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 -- start start 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: init 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc4009aa50 addr v1:172.21.15.5:6789/0 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc40071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc4013aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-b con 0x7fbc401386f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4009caa0 con 0x7fbc4009aa50 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4009e3a0 con 0x7fbc40071870 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4009ae30 con 0x7fbc401386f0 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 10 monclient(hunting): authenticate will time out at 1214.010539s 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46255700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fbc4009aa50 0x7fbc4009e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36442/0 (socket says 172.21.15.5:36442) 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46255700 1 -- 172.21.15.5:0/1126710244 learned_addr learned my addr 172.21.15.5:0/1126710244 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45a54700 1 -- 172.21.15.5:0/1126710244 >> v1:172.21.15.143:6789/0 conn(0x7fbc40071870 legacy=0x7fbc4009eff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc4013aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): get_auth_request con 0x7fbc401386f0 auth_method 0 2024-09-16T10:55:55.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919758605 0 0) 0x7fbc34001110 con 0x7fbc4009aa50 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): _negotiate 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): my global_id is 28773 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c0035d0 con 0x7fbc4009aa50 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc46a56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3446991911 0 0) 0x7fbc2c0035d0 con 0x7fbc4009aa50 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 msgr2=0x7fbc4013aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc4013aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 >> v1:172.21.15.143:6789/0 conn(0x7fbc40071870 legacy=0x7fbc4009eff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4009f700 con 0x7fbc4009aa50 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 5 monclient: authenticate success, global_id 28773 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7fbc34001800 con 0x7fbc4009aa50 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: dump: 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.448 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 >> v1:172.21.15.5:6789/0 conn(0x7fbc4009aa50 legacy=0x7fbc4009e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc2c003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): picked mon.c con 0x7fbc401386f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc2c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): picked mon.a con 0x7fbc2c003fb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc2c004db0 0x7fbc2c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): picked mon.b con 0x7fbc2c004db0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4009ae30 con 0x7fbc2c003fb0 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c00bbf0 con 0x7fbc2c004db0 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c00bdb0 con 0x7fbc401386f0 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45253700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:55:55.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc47257700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.431+0000 7fbc45a54700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc2c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46a56700 1 -- 172.21.15.5:0/1126710244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc2c004db0 msgr2=0x7fbc2c005230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): get_auth_request con 0x7fbc2c003fb0 auth_method 0 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46a56700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc2c004db0 0x7fbc2c005230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc2c003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): get_auth_request con 0x7fbc401386f0 auth_method 0 2024-09-16T10:55:55.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient(hunting): handle_auth_done global_id 28773 payload 293 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 -- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 msgr2=0x7fbc2c003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401386f0 0x7fbc2c003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 -- 172.21.15.5:0/1126710244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc2c004db0 msgr2=0x7fbc2c005230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc2c004db0 0x7fbc2c005230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 monclient: found mon.a 2024-09-16T10:55:55.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 -- 172.21.15.5:0/1126710244 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0035d0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45a54700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc2c004870 secure :-1 s=READY pgs=6854 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009dc0 tx=0x7fbc2c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5598cda1ef8fbd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38009860 con 0x7fbc2c003fb0 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: dump: 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.453 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3800bd60 con 0x7fbc2c003fb0 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 1 -- 172.21.15.5:0/1126710244 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc38015640 con 0x7fbc2c003fb0 2024-09-16T10:55:55.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 4 set_mon_vals no callback set 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc45253700 10 monclient: dump: 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.455 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 10 monclient: shutdown 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 msgr2=0x7fbc2c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 --2- 172.21.15.5:0/1126710244 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc2c004870 secure :-1 s=READY pgs=6854 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009dc0 tx=0x7fbc2c0058a0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc46a56700 1 -- 172.21.15.5:0/1126710244 reap_dead start 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- 172.21.15.5:0/1126710244 shutdown_connections 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- 172.21.15.5:0/1126710244 >> 172.21.15.5:0/1126710244 conn(0x7fbc400a45d0 msgr2=0x7fbc400a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- 172.21.15.5:0/1126710244 shutdown_connections 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- 172.21.15.5:0/1126710244 wait complete. 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 10 monclient: build_initial_monmap 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 10 monclient: monmap: 2024-09-16T10:55:55.456 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 librados: starting msgr at 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 librados: starting objecter 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 Processor -- start 2024-09-16T10:55:55.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 -- start start 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 librados: setting wanted keys 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 1 librados: calling monclient init 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.435+0000 7fbc47257700 10 monclient: init 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc401cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-a con 0x7fbc2c003fb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-c con 0x7fbc401cb060 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 0x7fbc4011c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): picked mon.noname-b con 0x7fbc401cb480 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4011f4b0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40130d10 con 0x7fbc401cb480 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4011e330 con 0x7fbc401cb060 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient(hunting): authenticate will time out at 1214.016492s 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46a56700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 msgr2=0x7fbc4011c3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc401cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46a56700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 0x7fbc4011c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44426/0 (socket says 172.21.15.5:44426) 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 1 -- 172.21.15.5:0/343678769 learned_addr learned my addr 172.21.15.5:0/343678769 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:55.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): get_auth_request con 0x7fbc401cb060 auth_method 0 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): get_auth_request con 0x7fbc2c003fb0 auth_method 0 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:55.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient(hunting): handle_auth_done global_id 28776 payload 824 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 -- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 msgr2=0x7fbc4011be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 -- 172.21.15.5:0/343678769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 msgr2=0x7fbc4011c3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 0x7fbc4011c3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient: _finish_hunting 0 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 monclient: found mon.noname-a 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 -- 172.21.15.5:0/343678769 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4011fa00 con 0x7fbc2c003fb0 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc45a54700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 client.?.objecter ms_handle_connect 0x7fbc2c003fb0 2024-09-16T10:55:55.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc46255700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc401cab20 secure :-1 s=READY pgs=6855 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007270 tx=0x7fbc34007580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34011910 con 0x7fbc2c003fb0 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: dump: 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.462 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: _finish_auth 0 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 5 monclient: authenticate success, global_id 28776 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _renew_subs 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34011a70 con 0x7fbc2c003fb0 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc400a2f90 con 0x7fbc2c003fb0 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34011e20 con 0x7fbc2c003fb0 2024-09-16T10:55:55.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc44a52700 4 set_mon_vals no callback set 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc32ffd700 10 monclient: dump: 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:55.464 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 client.28776.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _renew_subs 2024-09-16T10:55:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4009b120 con 0x7fbc2c003fb0 2024-09-16T10:55:55.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.439+0000 7fbc47257700 1 librados: init done 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fbc340103d0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc47257700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc47257700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc47257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4011f5f0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 1 --2- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 0x7fbc24039400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7fbc34048750 con 0x7fbc2c003fb0 2024-09-16T10:55:55.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 10 client.28776.objecter ms_dispatch 0x7fbc40000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 3 client.28776.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 3 client.28776.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 20 client.28776.objecter dump_active .. 0 homeless 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc45a54700 1 --2- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 0x7fbc24039400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc45a54700 10 monclient: get_auth_request con 0x7fbc24036f40 auth_method 0 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc45a54700 1 --2- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 0x7fbc24039400 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800c010 tx=0x7fbc38005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.443+0000 7fbc32ffd700 10 client.28776.objecter ms_handle_connect 0x7fbc24036f40 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.451+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbc4011f5f0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.451+0000 7fbc32ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:55.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.451+0000 7fbc32ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:55.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:55.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:55.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:55.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbc4009caa0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc32ffd700 1 -- 172.21.15.5:0/343678769 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fbc4009caa0 con 0x7fbc2c003fb0 2024-09-16T10:55:55.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc32ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:55.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc32ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:55.686 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:55.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 librados: watch_flush enter 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 librados: watch_flush exit 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 20 client.28776.objecter shutdown clearing up homeless session... 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 client.28776.objecter successfully canceled tick 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 msgr2=0x7fbc24039400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 0x7fbc24039400 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800c010 tx=0x7fbc38005000 comp rx=0 tx=0).stop 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 10 monclient: shutdown 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 msgr2=0x7fbc401cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc401cab20 secure :-1 s=READY pgs=6855 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007270 tx=0x7fbc34007580 comp rx=0 tx=0).stop 2024-09-16T10:55:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 shutdown_connections 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc401cb480 0x7fbc4011c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbc401cb060 0x7fbc4011be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbc24036f40 0x7fbc24039400 unknown :-1 s=CLOSED pgs=3786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 --2- 172.21.15.5:0/343678769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbc2c003fb0 0x7fbc401cab20 unknown :-1 s=CLOSED pgs=6855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 >> 172.21.15.5:0/343678769 conn(0x7fbc400a45d0 msgr2=0x7fbc40137f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 shutdown_connections 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 -- 172.21.15.5:0/343678769 wait complete. 2024-09-16T10:55:55.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:55.683+0000 7fbc47257700 1 librados: shutdown 2024-09-16T10:55:58.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:55:58.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: get_monmap_and_config 2024-09-16T10:55:58.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: build_initial_monmap 2024-09-16T10:55:58.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: monmap: 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:55:58.819111+0000 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:55:58.819111+0000 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:55:58.820 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:55:58.821 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 Processor -- start 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 -- start start 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: init 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T10:55:58.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f778009b4c0 addr v1:172.21.15.143:6789/0 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 0x7f778009bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): picked mon.noname-c con 0x7f778009dc50 addr v2:172.21.15.143:3300/0 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): picked mon.noname-b con 0x7f7780137c50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778009cf40 con 0x7f778009b4c0 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800a0850 con 0x7f778009dc50 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778009ccf0 con 0x7f7780137c50 2024-09-16T10:55:58.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7786896700 10 monclient(hunting): authenticate will time out at 1217.395670s 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f777f7fe700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 msgr2=0x7f778009bfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f777ffff700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f778009b4c0 legacy=0x7f778009b8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f777f7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 0x7f778009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44440/0 (socket says 172.21.15.5:44440) 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 1 -- 172.21.15.5:0/1678309296 learned_addr learned my addr 172.21.15.5:0/1678309296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 10 monclient(hunting): get_auth_request con 0x7f7780137c50 auth_method 0 2024-09-16T10:55:58.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.815+0000 7f7784e33700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient(hunting): handle_auth_done global_id 54179 payload 274 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 -- 172.21.15.5:0/1678309296 >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 msgr2=0x7f778009bfb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 --2- 172.21.15.5:0/1678309296 >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 0x7f778009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:58.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 -- 172.21.15.5:0/1678309296 >> v1:172.21.15.143:6789/0 conn(0x7f778009b4c0 legacy=0x7f778009b8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient: _finish_hunting 0 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 monclient: found mon.noname-b 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 -- 172.21.15.5:0/1678309296 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778009aa50 con 0x7f7780137c50 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7784e33700 1 --2- 172.21.15.5:0/1678309296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e80 tx=0x7f777800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a91b16d4be861193 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 1 -- 172.21.15.5:0/1678309296 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778003c40 con 0x7f7780137c50 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: dump: 2024-09-16T10:55:58.834 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:58.835 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: _finish_auth 0 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: _check_auth_tickets 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 5 monclient: authenticate success, global_id 54179 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 1 -- 172.21.15.5:0/1678309296 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7778003da0 con 0x7f7780137c50 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: get_monmap_and_config success 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 4 set_mon_vals no callback set 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 1 -- 172.21.15.5:0/1678309296 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777800bb80 con 0x7f7780137c50 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:55:58.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777effd700 10 monclient: dump: 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: shutdown 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:58.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- 172.21.15.5:0/1678309296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 msgr2=0x7f778013a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- 172.21.15.5:0/1678309296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7778009e80 tx=0x7f777800b330 comp rx=0 tx=0).stop 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- 172.21.15.5:0/1678309296 shutdown_connections 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- 172.21.15.5:0/1678309296 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7780137c50 0x7f778013a040 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- 172.21.15.5:0/1678309296 >> v2:172.21.15.143:3300/0 conn(0x7f778009dc50 0x7f778009bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- 172.21.15.5:0/1678309296 >> 172.21.15.5:0/1678309296 conn(0x7f77800a0b30 msgr2=0x7f77800a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- 172.21.15.5:0/1678309296 shutdown_connections 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- 172.21.15.5:0/1678309296 wait complete. 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: build_initial_monmap 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: monmap: 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:55:58.838 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 librados: starting msgr at 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 librados: starting objecter 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 Processor -- start 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- start start 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 librados: setting wanted keys 2024-09-16T10:55:58.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 librados: calling monclient init 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: init 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: _reopen_session rank -1 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 0x7f778006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): picked mon.noname-c con 0x7f778009b4c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): picked mon.noname-a con 0x7f778009dc50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 0x7f778006f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): picked mon.noname-b con 0x7f7780137c50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780070c00 con 0x7f778009dc50 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:58.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77800a3520 con 0x7f7780137c50 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): start opening mon connection 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778011fcc0 con 0x7f778009b4c0 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): _renew_subs 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f7786896700 10 monclient(hunting): authenticate will time out at 1217.399852s 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 0x7f778006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.819+0000 7f777f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59980/0 (socket says 172.21.15.5:59980) 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 -- 172.21.15.5:0/2271526902 learned_addr learned my addr 172.21.15.5:0/2271526902 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): get_auth_request con 0x7f778009dc50 auth_method 0 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:58.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7784e33700 1 -- 172.21.15.5:0/2271526902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 msgr2=0x7f778006f570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): get_auth_request con 0x7f778009b4c0 auth_method 0 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7784e33700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 0x7f778006f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient(hunting): handle_auth_done global_id 28779 payload 824 2024-09-16T10:55:58.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 -- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 msgr2=0x7f778006eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 0x7f778006eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 -- 172.21.15.5:0/2271526902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 msgr2=0x7f778006f570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 0x7f778006f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 monclient: found mon.noname-a 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 -- 172.21.15.5:0/2271526902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778009b30 con 0x7f778009dc50 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 0x7f778006eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777f7fe700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 secure :-1 s=READY pgs=6861 cs=0 l=1 rev1=1 crypto rx=0x7f777400b780 tx=0x7f777400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 client.?.objecter ms_handle_connect 0x7f778009dc50 2024-09-16T10:55:58.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774003e70 con 0x7f778009dc50 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: dump: 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:58.844 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: _finish_auth 0 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: _check_auth_tickets 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774005070 con 0x7f778009dc50 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 5 monclient: authenticate success, global_id 28779 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774005420 con 0x7f778009dc50 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _renew_subs 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77800a2ff0 con 0x7f778009dc50 2024-09-16T10:55:58.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777e7fc700 4 set_mon_vals no callback set 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 monclient: dump: 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:55:58.846 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 client.28779.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _renew_subs 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778011f9e0 con 0x7f778009dc50 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 1 librados: init done 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778011f420 con 0x7f778009dc50 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7774005580 con 0x7f778009dc50 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 --2- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 0x7f776c0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:55:58.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f7774047b70 con 0x7f778009dc50 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 10 client.28779.objecter ms_dispatch 0x7f7780000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 3 client.28779.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 3 client.28779.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777cff9700 20 client.28779.objecter dump_active .. 0 homeless 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 1 --2- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 0x7f776c0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.823+0000 7f777ffff700 10 monclient: get_auth_request con 0x7f776c036f10 auth_method 0 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.827+0000 7f777ffff700 1 --2- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 0x7f776c0393d0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7f77700039e0 tx=0x7f7770003c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:55:58.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.827+0000 7f777cff9700 10 client.28779.objecter ms_handle_connect 0x7f776c036f10 2024-09-16T10:55:58.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.831+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7774005830 con 0x7f778009dc50 2024-09-16T10:55:58.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.831+0000 7f777cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:55:58.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:58.831+0000 7f777cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:55:59.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f7786896700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f7786896700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f7786896700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f778009b8a0 con 0x7f778009dc50 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f777cff9700 1 -- 172.21.15.5:0/2271526902 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f7774019360 con 0x7f778009dc50 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f777cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:55:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.067+0000 7f777cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:55:59.073 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:55:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 10 librados: watch_flush enter 2024-09-16T10:55:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 10 librados: watch_flush exit 2024-09-16T10:55:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 20 client.28779.objecter shutdown clearing up homeless session... 2024-09-16T10:55:59.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 10 client.28779.objecter successfully canceled tick 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 msgr2=0x7f776c0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 0x7f776c0393d0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7f77700039e0 tx=0x7f7770003c40 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 10 monclient: shutdown 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 msgr2=0x7f778006f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 secure :-1 s=READY pgs=6861 cs=0 l=1 rev1=1 crypto rx=0x7f777400b780 tx=0x7f777400ba90 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 shutdown_connections 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7780137c50 0x7f778006f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f776c036f10 0x7f776c0393d0 unknown :-1 s=CLOSED pgs=3787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f778009dc50 0x7f778006f030 unknown :-1 s=CLOSED pgs=6861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 --2- 172.21.15.5:0/2271526902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f778009b4c0 0x7f778006eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 >> 172.21.15.5:0/2271526902 conn(0x7f77800a0b30 msgr2=0x7f778009a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:55:59.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 shutdown_connections 2024-09-16T10:55:59.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 -- 172.21.15.5:0/2271526902 wait complete. 2024-09-16T10:55:59.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:55:59.071+0000 7f7786896700 1 librados: shutdown 2024-09-16T10:56:02.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:56:02.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: get_monmap_and_config 2024-09-16T10:56:02.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: build_initial_monmap 2024-09-16T10:56:02.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: monmap: 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:56:02.209110+0000 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:56:02.209110+0000 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:56:02.210 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 Processor -- start 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 -- start start 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: init 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f2210070ad0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f221009dc30 addr v1:172.21.15.5:6789/0 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22100713f0 addr v1:172.21.15.143:6789/0 2024-09-16T10:56:02.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221009e0b0 con 0x7f221009dc30 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22100a2810 con 0x7f22100713f0 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221009f150 con 0x7f2210070ad0 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient(hunting): authenticate will time out at 1220.785906s 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220f7fe700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f22100713f0 legacy=0x7f22100a2100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:44450/0 (socket says 172.21.15.5:44450) 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220e7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f221009dc30 0x7f22100a19f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36450/0 (socket says 172.21.15.5:36450) 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 -- 172.21.15.5:0/182078244 learned_addr learned my addr 172.21.15.5:0/182078244 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): get_auth_request con 0x7f2210070ad0 auth_method 0 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:02.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient(hunting): handle_auth_done global_id 54185 payload 274 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 -- 172.21.15.5:0/182078244 >> v1:172.21.15.143:6789/0 conn(0x7f22100713f0 legacy=0x7f22100a2100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 -- 172.21.15.5:0/182078244 >> v1:172.21.15.5:6789/0 conn(0x7f221009dc30 legacy=0x7f22100a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient: _finish_hunting 0 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 monclient: found mon.noname-b 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:56:02.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 -- 172.21.15.5:0/182078244 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221009f9f0 con 0x7f2210070ad0 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220effd700 1 --2- 172.21.15.5:0/182078244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2200009940 tx=0x7f2200009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=833704cb155e5ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 1 -- 172.21.15.5:0/182078244 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200004230 con 0x7f2210070ad0 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: dump: 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:02.217 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:02.218 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:02.218 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: _finish_auth 0 2024-09-16T10:56:02.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:56:02.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 5 monclient: authenticate success, global_id 54185 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 1 -- 172.21.15.5:0/182078244 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200009e90 con 0x7f2210070ad0 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 10 monclient: get_monmap_and_config success 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220ffff700 4 set_mon_vals no callback set 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 1 -- 172.21.15.5:0/182078244 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000045b0 con 0x7f2210070ad0 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.206+0000 7f220dffb700 10 monclient: dump: 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:02.219 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: shutdown 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- 172.21.15.5:0/182078244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 msgr2=0x7f2210070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 --2- 172.21.15.5:0/182078244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2200009940 tx=0x7f2200009c50 comp rx=0 tx=0).stop 2024-09-16T10:56:02.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- 172.21.15.5:0/182078244 shutdown_connections 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 --2- 172.21.15.5:0/182078244 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2210070ad0 0x7f2210070eb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- 172.21.15.5:0/182078244 >> 172.21.15.5:0/182078244 conn(0x7f22100a3200 msgr2=0x7f2210073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- 172.21.15.5:0/182078244 shutdown_connections 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- 172.21.15.5:0/182078244 wait complete. 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: build_initial_monmap 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: monmap: 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:56:02.221 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:56:02.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:56:02.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 librados: starting msgr at 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 librados: starting objecter 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 Processor -- start 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- start start 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 librados: setting wanted keys 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 librados: calling monclient init 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: init 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f2210070ad0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:56:02.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22100713f0 0x7f22101c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f22100713f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 0x7f22101c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f22101c83c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22101a94d0 con 0x7f2210070ad0 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22100a55b0 con 0x7f22101c83c0 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2210070330 con 0x7f22100713f0 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220ffff700 10 monclient(hunting): authenticate will time out at 1220.790597s 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220f7fe700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 msgr2=0x7f22101c8850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:02.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 0x7f22101c8850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59994/0 (socket says 172.21.15.5:59994) 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 -- 172.21.15.5:0/954125411 learned_addr learned my addr 172.21.15.5:0/954125411 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): get_auth_request con 0x7f2210070ad0 auth_method 0 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22100713f0 0x7f22101c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): get_auth_request con 0x7f22100713f0 auth_method 0 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:02.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient(hunting): handle_auth_done global_id 28782 payload 824 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 -- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22100713f0 msgr2=0x7f22101c2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22100713f0 0x7f22101c2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 -- 172.21.15.5:0/954125411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 msgr2=0x7f22101c8850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 0x7f22101c8850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient: _finish_hunting 0 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 monclient: found mon.noname-a 2024-09-16T10:56:02.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.210+0000 7f220effd700 1 -- 172.21.15.5:0/954125411 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200009570 con 0x7f2210070ad0 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 client.?.objecter ms_handle_connect 0x7f2210070ad0 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220effd700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 secure :-1 s=READY pgs=6864 cs=0 l=1 rev1=1 crypto rx=0x7f2200009150 tx=0x7f2200004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000036c0 con 0x7f2210070ad0 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: dump: 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:02.231 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: _finish_auth 0 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: _check_auth_tickets 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 5 monclient: authenticate success, global_id 28782 2024-09-16T10:56:02.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200003820 con 0x7f2210070ad0 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _renew_subs 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221009d1d0 con 0x7f2210070ad0 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200003b90 con 0x7f2210070ad0 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220d7fa700 4 set_mon_vals no callback set 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 monclient: dump: 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 client.28782.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _renew_subs 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22101a9d70 con 0x7f2210070ad0 2024-09-16T10:56:02.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 1 librados: init done 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2200021070 con 0x7f2210070ad0 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 --2- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 0x7f21f40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f22000499c0 con 0x7f2210070ad0 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221009d030 con 0x7f2210070ad0 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 10 client.28782.objecter ms_dispatch 0x7f2210000fd0 osd_map(275..275 src has 1..275) v4 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 3 client.28782.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 3 client.28782.objecter handle_osd_map decoding full epoch 275 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f22077fe700 20 client.28782.objecter dump_active .. 0 homeless 2024-09-16T10:56:02.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220e7fc700 1 --2- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 0x7f21f40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.214+0000 7f220e7fc700 10 monclient: get_auth_request con 0x7f21f4036f10 auth_method 0 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.218+0000 7f22077fe700 10 client.28782.objecter ms_handle_connect 0x7f21f4036f10 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.218+0000 7f220e7fc700 1 --2- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 0x7f21f40393d0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7f22101a7e20 tx=0x7f21f8001b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.222+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f221009d030 con 0x7f2210070ad0 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.222+0000 7f22077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:02.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.222+0000 7f22077fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:56:02.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f220ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:56:02.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f220ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:56:02.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f220ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:02.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f221009d030 con 0x7f2210070ad0 2024-09-16T10:56:02.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f22077fe700 1 -- 172.21.15.5:0/954125411 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7f221009d030 con 0x7f2210070ad0 2024-09-16T10:56:02.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f22077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:56:02.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.454+0000 7f22077fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:56:02.459 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:56:02.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 10 librados: watch_flush enter 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 10 librados: watch_flush exit 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 20 client.28782.objecter shutdown clearing up homeless session... 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 10 client.28782.objecter successfully canceled tick 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 msgr2=0x7f21f40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 0x7f21f40393d0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7f22101a7e20 tx=0x7f21f8001b20 comp rx=0 tx=0).stop 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 10 monclient: shutdown 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 msgr2=0x7f22101c4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 secure :-1 s=READY pgs=6864 cs=0 l=1 rev1=1 crypto rx=0x7f2200009150 tx=0x7f2200004770 comp rx=0 tx=0).stop 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 shutdown_connections 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22101c83c0 0x7f22101c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f22100713f0 0x7f22101c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> v2:172.21.15.143:6812/1098205034 conn(0x7f21f4036f10 0x7f21f40393d0 unknown :-1 s=CLOSED pgs=3788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 --2- 172.21.15.5:0/954125411 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2210070ad0 0x7f22101c4e80 unknown :-1 s=CLOSED pgs=6864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 >> 172.21.15.5:0/954125411 conn(0x7f22100a3200 msgr2=0x7f221006e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 shutdown_connections 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 -- 172.21.15.5:0/954125411 wait complete. 2024-09-16T10:56:02.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:02.458+0000 7f220ffff700 1 librados: shutdown 2024-09-16T10:56:05.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:56:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: get_monmap_and_config 2024-09-16T10:56:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: build_initial_monmap 2024-09-16T10:56:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: monmap: 2024-09-16T10:56:05.595 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:56:05.594951+0000 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:56:05.594951+0000 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:56:05.596 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 Processor -- start 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 -- start start 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: init 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f775c0a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f775c09f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): picked mon.noname-b con 0x7f775c0a36e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c0a3c40 con 0x7f775c09f780 2024-09-16T10:56:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c0a4300 con 0x7f775c0a2bf0 2024-09-16T10:56:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c09fda0 con 0x7f775c0a36e0 2024-09-16T10:56:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7762515700 10 monclient(hunting): authenticate will time out at 1224.171471s 2024-09-16T10:56:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761513700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f775c0a2bf0 legacy=0x7f775c0a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7760d12700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f775c09f780 0x7f775c13ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:32784/0 (socket says 172.21.15.5:32784) 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 learned_addr learned my addr 172.21.15.5:0/1161506238 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761d14700 10 monclient(hunting): get_auth_request con 0x7f775c0a36e0 auth_method 0 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761d14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761d14700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.590+0000 7f7761d14700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.594+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:56:05.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.594+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.594+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.594+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:05.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.794+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v1:172.21.15.143:6789/0 conn(0x7f775c0a2bf0 legacy=0x7f775c0a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:05.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:05.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 10 monclient(hunting): get_auth_request con 0x7f775c0a36e0 auth_method 0 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:05.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:05.798+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:56:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.198+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v1:172.21.15.143:6789/0 conn(0x7f775c0a2bf0 legacy=0x7f775c0a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:06.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 10 monclient(hunting): get_auth_request con 0x7f775c0a36e0 auth_method 0 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:06.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:06.202+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:56:07.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.002+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v1:172.21.15.143:6789/0 conn(0x7f775c0a2bf0 legacy=0x7f775c0a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:07.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 10 monclient(hunting): get_auth_request con 0x7f775c0a36e0 auth_method 0 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:07.006+0000 7f7761d14700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): tick 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 monclient(hunting): continuing hunt 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 msgr2=0x7f775c13e530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c0a36e0 0x7f775c13e530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:08.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 -- 172.21.15.5:0/1161506238 >> v1:172.21.15.143:6789/0 conn(0x7f775c0a2bf0 legacy=0x7f775c0a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 -- 172.21.15.5:0/1161506238 >> v1:172.21.15.5:6789/0 conn(0x7f775c09f780 legacy=0x7f775c13ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient: _add_conns ranks=[4,3] 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): picked mon.noname-c con 0x7f7740003750 addr v2:172.21.15.143:3300/0 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f7740006220 addr v2:172.21.15.5:3300/0 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 -- 172.21.15.5:0/1161506238 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c09fda0 con 0x7f7740006220 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 1 -- 172.21.15.5:0/1161506238 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7740008d90 con 0x7f7740003750 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7752ffd700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:08.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 msgr2=0x7f7740005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7761513700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 10 monclient(hunting): get_auth_request con 0x7f7740006220 auth_method 0 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:08.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.594+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 msgr2=0x7f7740005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7761513700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 10 monclient(hunting): get_auth_request con 0x7f7740006220 auth_method 0 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:08.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:08.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:08.798+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:56:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 msgr2=0x7f7740005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7761513700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 10 monclient(hunting): get_auth_request con 0x7f7740006220 auth_method 0 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 msgr2=0x7f77400086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:56:09.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:09.202+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7761513700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 msgr2=0x7f7740005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.5:3300/0 conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7761513700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): get_auth_request con 0x7f7740006220 auth_method 0 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient(hunting): handle_auth_done global_id 28785 payload 274 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 msgr2=0x7f7740005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> v2:172.21.15.143:3300/0 conn(0x7f7740003750 0x7f7740005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 monclient: found mon.noname-a 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 -- 172.21.15.5:0/1161506238 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c0a0080 con 0x7f7740006220 2024-09-16T10:56:10.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7760d12700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f77400086e0 secure :-1 s=READY pgs=6867 cs=0 l=1 rev1=1 crypto rx=0x7f77400036e0 tx=0x7f7758007330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5956c7ea3b642b86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 1 -- 172.21.15.5:0/1161506238 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758005e10 con 0x7f7740006220 2024-09-16T10:56:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:56:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: dump: 2024-09-16T10:56:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.016 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 5 monclient: authenticate success, global_id 28785 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 1 -- 172.21.15.5:0/1161506238 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758016480 con 0x7f7740006220 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 10 monclient: get_monmap_and_config success 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 1 -- 172.21.15.5:0/1161506238 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758016810 con 0x7f7740006220 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 4 set_mon_vals no callback set 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7753fff700 10 monclient: dump: 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.019 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 10 monclient: shutdown 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 -- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 msgr2=0x7f77400086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f77400086e0 secure :-1 s=READY pgs=6867 cs=0 l=1 rev1=1 crypto rx=0x7f77400036e0 tx=0x7f7758007330 comp rx=0 tx=0).stop 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7761d14700 1 -- 172.21.15.5:0/1161506238 reap_dead start 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 -- 172.21.15.5:0/1161506238 shutdown_connections 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 --2- 172.21.15.5:0/1161506238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f77400086e0 unknown :-1 s=CLOSED pgs=6867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 -- 172.21.15.5:0/1161506238 >> 172.21.15.5:0/1161506238 conn(0x7f775c0a53e0 msgr2=0x7f775c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 -- 172.21.15.5:0/1161506238 shutdown_connections 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.006+0000 7f7762515700 1 -- 172.21.15.5:0/1161506238 wait complete. 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient: build_initial_monmap 2024-09-16T10:56:10.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient: monmap: 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 librados: starting msgr at 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 librados: starting objecter 2024-09-16T10:56:10.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 Processor -- start 2024-09-16T10:56:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 -- start start 2024-09-16T10:56:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 librados: setting wanted keys 2024-09-16T10:56:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 librados: calling monclient init 2024-09-16T10:56:10.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient: init 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): picked mon.noname-a con 0x7f7740006220 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c135880 0x7f775c13aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): picked mon.noname-c con 0x7f775c135880 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 0x7f775c13afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): picked mon.noname-b con 0x7f775c135d00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c11ebe0 con 0x7f7740006220 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c136170 con 0x7f775c135d00 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c13d520 con 0x7f775c135880 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7762515700 10 monclient(hunting): authenticate will time out at 1228.591732s 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7761d14700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 msgr2=0x7f775c13afb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.010+0000 7f7761513700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c135880 0x7f775c13aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761d14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 0x7f775c13afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51322/0 (socket says 172.21.15.5:51322) 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 -- 172.21.15.5:0/2131217198 learned_addr learned my addr 172.21.15.5:0/2131217198 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): get_auth_request con 0x7f7740006220 auth_method 0 2024-09-16T10:56:10.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): get_auth_request con 0x7f775c135880 auth_method 0 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient(hunting): handle_auth_done global_id 28788 payload 824 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 -- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c135880 msgr2=0x7f775c13aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c135880 0x7f775c13aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 -- 172.21.15.5:0/2131217198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 msgr2=0x7f775c13afb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 0x7f775c13afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 monclient: found mon.noname-a 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 -- 172.21.15.5:0/2131217198 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775800f040 con 0x7f7740006220 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 client.?.objecter ms_handle_connect 0x7f7740006220 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7761513700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 secure :-1 s=READY pgs=6868 cs=0 l=1 rev1=1 crypto rx=0x7f774c003e20 tx=0x7f774c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c00c980 con 0x7f7740006220 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: dump: 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.030 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 5 monclient: authenticate success, global_id 28788 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c00a040 con 0x7f7740006220 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _renew_subs 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c09f4e0 con 0x7f7740006220 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c00cd40 con 0x7f7740006220 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 monclient: dump: 2024-09-16T10:56:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f77537fe700 4 set_mon_vals no callback set 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 client.28788.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _renew_subs 2024-09-16T10:56:10.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c0a7670 con 0x7f7740006220 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 1 librados: init done 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f774c012030 con 0x7f7740006220 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c0a2fd0 con 0x7f7740006220 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 --2- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 0x7f774003a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f774c047530 con 0x7f7740006220 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 10 client.28788.objecter ms_dispatch 0x7f775c000fd0 osd_map(276..276 src has 1..276) v4 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 3 client.28788.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 3 client.28788.objecter handle_osd_map decoding full epoch 276 2024-09-16T10:56:10.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7751ffb700 20 client.28788.objecter dump_active .. 0 homeless 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 1 --2- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 0x7f774003a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.014+0000 7f7760d12700 10 monclient: get_auth_request con 0x7f7740037c30 auth_method 0 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.018+0000 7f7751ffb700 10 client.28788.objecter ms_handle_connect 0x7f7740037c30 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.018+0000 7f7760d12700 1 --2- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 0x7f774003a0f0 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7f7758000c00 tx=0x7f7758024040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.026+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f774c011690 con 0x7f7740006220 2024-09-16T10:56:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.026+0000 7f7751ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.026+0000 7f7751ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:56:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.258+0000 7f7762515700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:56:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.258+0000 7f7762515700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:56:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.258+0000 7f7762515700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.258+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f775c13d010 con 0x7f7740006220 2024-09-16T10:56:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.262+0000 7f7751ffb700 1 -- 172.21.15.5:0/2131217198 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f774c01c300 con 0x7f7740006220 2024-09-16T10:56:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.262+0000 7f7751ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:56:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.262+0000 7f7751ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:56:10.266 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.262+0000 7f7762515700 10 librados: watch_flush enter 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 10 librados: watch_flush exit 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 20 client.28788.objecter shutdown clearing up homeless session... 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 10 client.28788.objecter successfully canceled tick 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 msgr2=0x7f774003a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 0x7f774003a0f0 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7f7758000c00 tx=0x7f7758024040 comp rx=0 tx=0).stop 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 10 monclient: shutdown 2024-09-16T10:56:10.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 msgr2=0x7f775c135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 secure :-1 s=READY pgs=6868 cs=0 l=1 rev1=1 crypto rx=0x7f774c003e20 tx=0x7f774c00bc00 comp rx=0 tx=0).stop 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 shutdown_connections 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f775c135d00 0x7f775c13afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f775c135880 0x7f775c13aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7740037c30 0x7f774003a0f0 unknown :-1 s=CLOSED pgs=3790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 --2- 172.21.15.5:0/2131217198 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7740006220 0x7f775c135340 unknown :-1 s=CLOSED pgs=6868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 >> 172.21.15.5:0/2131217198 conn(0x7f775c0a53e0 msgr2=0x7f775c0a68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 shutdown_connections 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 -- 172.21.15.5:0/2131217198 wait complete. 2024-09-16T10:56:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.266+0000 7f7762515700 1 librados: shutdown 2024-09-16T10:56:10.287 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T10:56:10.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T10:56:10.287 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:56:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.394+0000 7f640359e700 10 monclient: get_monmap_and_config 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.394+0000 7f640359e700 10 monclient: build_initial_monmap 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.394+0000 7f640359e700 10 monclient: monmap: 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:56:10.400743+0000 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:56:10.400743+0000 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:56:10.402 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:56:10.403 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:56:10.403 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 Processor -- start 2024-09-16T10:56:10.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- start start 2024-09-16T10:56:10.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient: init 2024-09-16T10:56:10.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:10.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T10:56:10.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6404070750 addr v1:172.21.15.5:6789/0 2024-09-16T10:56:10.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64040a04b0 addr v1:172.21.15.143:6789/0 2024-09-16T10:56:10.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 0x7f640413e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): picked mon.noname-b con 0x7f64040a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640409fb60 con 0x7f6404070750 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040a0090 con 0x7f64040a04b0 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640409f7f0 con 0x7f64040a14c0 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient(hunting): authenticate will time out at 1228.977456s 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640259c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6404070750 0x7f6404070b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:32794/0 (socket says 172.21.15.5:32794) 2024-09-16T10:56:10.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640259c700 1 -- 172.21.15.5:0/4114355757 learned_addr learned my addr 172.21.15.5:0/4114355757 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6401d9b700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f64040a04b0 legacy=0x7f640413ad80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 --2- 172.21.15.5:0/4114355757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 0x7f640413e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): get_auth_request con 0x7f64040a14c0 auth_method 0 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient(hunting): handle_auth_done global_id 54194 payload 274 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 -- 172.21.15.5:0/4114355757 >> v1:172.21.15.143:6789/0 conn(0x7f64040a04b0 legacy=0x7f640413ad80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:10.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 1 -- 172.21.15.5:0/4114355757 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023156331 0 0) 0x7f640409fb60 con 0x7f6404070750 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 -- 172.21.15.5:0/4114355757 >> v1:172.21.15.5:6789/0 conn(0x7f6404070750 legacy=0x7f6404070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 monclient: found mon.noname-b 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 -- 172.21.15.5:0/4114355757 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640409ad40 con 0x7f64040a14c0 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f6402d9d700 1 --2- 172.21.15.5:0/4114355757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 0x7f640413e4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009dc0 tx=0x7f63fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fc0f998cdb79d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 1 -- 172.21.15.5:0/4114355757 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00e070 con 0x7f64040a14c0 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:56:10.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: dump: 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.410 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 1 -- 172.21.15.5:0/4114355757 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00bb90 con 0x7f64040a14c0 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 5 monclient: authenticate success, global_id 54194 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 1 -- 172.21.15.5:0/4114355757 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003a40 con 0x7f64040a14c0 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient: get_monmap_and_config success 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 4 set_mon_vals no callback set 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640159a700 10 monclient: dump: 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 10 monclient: shutdown 2024-09-16T10:56:10.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- 172.21.15.5:0/4114355757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 msgr2=0x7f640413e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 --2- 172.21.15.5:0/4114355757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 0x7f640413e4b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009dc0 tx=0x7f63fc00b330 comp rx=0 tx=0).stop 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- 172.21.15.5:0/4114355757 shutdown_connections 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 --2- 172.21.15.5:0/4114355757 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f64040a14c0 0x7f640413e4b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- 172.21.15.5:0/4114355757 >> 172.21.15.5:0/4114355757 conn(0x7f64040a45d0 msgr2=0x7f64040a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.398+0000 7f640359e700 1 -- 172.21.15.5:0/4114355757 shutdown_connections 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 -- 172.21.15.5:0/4114355757 wait complete. 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient: build_initial_monmap 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient: monmap: 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:56:10.413 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:56:10.414 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 librados: starting msgr at 2024-09-16T10:56:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 librados: starting objecter 2024-09-16T10:56:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 Processor -- start 2024-09-16T10:56:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 -- start start 2024-09-16T10:56:10.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 librados: setting wanted keys 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 librados: calling monclient init 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient: init 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 0x7f64041ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): picked mon.noname-c con 0x7f6404070750 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64040a04b0 0x7f6404134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): picked mon.noname-a con 0x7f64040a04b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64040a14c0 0x7f6404135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): picked mon.noname-b con 0x7f64040a14c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404121c30 con 0x7f64040a04b0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404120170 con 0x7f64040a14c0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640409b0d0 con 0x7f6404070750 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640359e700 10 monclient(hunting): authenticate will time out at 1228.983759s 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 0x7f64041ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 0x7f64041ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:35852/0 (socket says 172.21.15.5:35852) 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f6401d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64040a04b0 0x7f6404134b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f6402d9d700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64040a14c0 msgr2=0x7f6404135050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 1 -- 172.21.15.5:0/2798793282 learned_addr learned my addr 172.21.15.5:0/2798793282 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 10 monclient(hunting): get_auth_request con 0x7f6404070750 auth_method 0 2024-09-16T10:56:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f640259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.402+0000 7f6402d9d700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64040a14c0 0x7f6404135050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): get_auth_request con 0x7f64040a04b0 auth_method 0 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient(hunting): handle_auth_done global_id 54197 payload 824 2024-09-16T10:56:10.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64040a14c0 msgr2=0x7f6404135050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64040a14c0 0x7f6404135050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64040a04b0 msgr2=0x7f6404134b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f64040a04b0 0x7f6404134b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 monclient: found mon.noname-c 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc009a70 con 0x7f6404070750 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640259c700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 0x7f64041ca3d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c5b0 tx=0x7f63f800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 client.?.objecter ms_handle_connect 0x7f6404070750 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8004ca0 con 0x7f6404070750 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: dump: 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 5 monclient: authenticate success, global_id 54197 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8004e20 con 0x7f6404070750 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 monclient: _renew_subs 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64041224d0 con 0x7f6404070750 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8007310 con 0x7f6404070750 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6400d99700 4 set_mon_vals no callback set 2024-09-16T10:56:10.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 monclient: dump: 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 client.54197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 monclient: _renew_subs 2024-09-16T10:56:10.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64040a24e0 con 0x7f6404070750 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 1 librados: init done 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 0x7f640411e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404067380 con 0x7f6404071240 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 0x7f640411e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f63f8007470 con 0x7f6404070750 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient: get_auth_request con 0x7f6404071240 auth_method 0 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 --2- 172.21.15.5:0/2798793282 >> v2:172.21.15.143:6812/1098205034 conn(0x7f63e4036f10 0x7f63e40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 10 monclient(hunting): handle_auth_done global_id 28797 payload 824 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6404070b30 con 0x7f6404071240 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f63f8047e70 con 0x7f6404070750 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 client.54197.objecter ms_dispatch 0x7f6404000fd0 osd_map(276..276 src has 1..276) v4 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 3 client.54197.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 3 client.54197.objecter handle_osd_map decoding full epoch 276 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6401d9b700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 0x7f640411e520 secure :-1 s=READY pgs=6870 cs=0 l=1 rev1=1 crypto rx=0x7f640411fd70 tx=0x7f63f4009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 20 client.54197.objecter dump_active .. 0 homeless 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 client.54197.objecter ms_handle_connect 0x7f6404071240 2024-09-16T10:56:10.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f63f2ffd700 10 client.54197.objecter resend_mon_ops 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.406+0000 7f6402d9d700 1 --2- 172.21.15.5:0/2798793282 >> v2:172.21.15.143:6812/1098205034 conn(0x7f63e4036f10 0x7f63e40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f6402d9d700 10 monclient: get_auth_request con 0x7f63e4036f10 auth_method 0 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4012600 con 0x7f6404071240 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 monclient: dump: 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f63f4012760 con 0x7f6404071240 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 msgr2=0x7f640411e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 0x7f640411e520 secure :-1 s=READY pgs=6870 cs=0 l=1 rev1=1 crypto rx=0x7f640411fd70 tx=0x7f63f4009aa0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 msgr2=0x7f640411e520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6404071240 0x7f640411e520 unknown :-1 s=CLOSED pgs=6870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f63f2ffd700 10 client.54197.objecter ms_handle_connect 0x7f63e4036f10 2024-09-16T10:56:10.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.410+0000 7f6402d9d700 1 --2- 172.21.15.5:0/2798793282 >> v2:172.21.15.143:6812/1098205034 conn(0x7f63e4036f10 0x7f63e40393d0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f64040a18d0 tx=0x7f63fc011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 0x7f640411ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640411f3b0 con 0x7f640411ea60 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 0x7f640411ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient: get_auth_request con 0x7f640411ea60 auth_method 0 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 10 monclient(hunting): handle_auth_done global_id 28800 payload 824 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 1 -- 172.21.15.5:0/2798793282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6404120ac0 con 0x7f640411ea60 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6401d9b700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 0x7f640411ee70 secure :-1 s=READY pgs=6871 cs=0 l=1 rev1=1 crypto rx=0x7f640411fd70 tx=0x7f63f4005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 client.54197.objecter ms_handle_connect 0x7f640411ea60 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 client.54197.objecter resend_mon_ops 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4023070 con 0x7f640411ea60 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 monclient: dump: 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.436 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7f63f401ee80 con 0x7f640411ea60 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 msgr2=0x7f640411ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 0x7f640411ee70 secure :-1 s=READY pgs=6871 cs=0 l=1 rev1=1 crypto rx=0x7f640411fd70 tx=0x7f63f4005a80 comp rx=0 tx=0).stop 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 msgr2=0x7f640411ee70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f63f2ffd700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f640411ea60 0x7f640411ee70 secure :-1 s=CLOSED pgs=6871 cs=0 l=1 rev1=1 crypto rx=0x7f640411fd70 tx=0x7f63f4005a80 comp rx=0 tx=0).stop 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 34, 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:56:10.438 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.439 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.440 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:56:10.441 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.442 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:56:10.443 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.444 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.445 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:56:10.446 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.447 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.448 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.449 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:56:10.450 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 librados: watch_flush enter 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 librados: watch_flush exit 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 20 client.54197.objecter shutdown clearing up homeless session... 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 client.54197.objecter successfully canceled tick 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 >> v2:172.21.15.143:6812/1098205034 conn(0x7f63e4036f10 msgr2=0x7f63e40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 --2- 172.21.15.5:0/2798793282 >> v2:172.21.15.143:6812/1098205034 conn(0x7f63e4036f10 0x7f63e40393d0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f64040a18d0 tx=0x7f63fc011000 comp rx=0 tx=0).stop 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 10 monclient: shutdown 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 msgr2=0x7f64041ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 --2- 172.21.15.5:0/2798793282 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6404070750 0x7f64041ca3d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c5b0 tx=0x7f63f800c8c0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6402d9d700 1 -- 172.21.15.5:0/2798793282 reap_dead start 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f6402d9d700 1 -- 172.21.15.5:0/2798793282 reap_dead start 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 shutdown_connections 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 >> 172.21.15.5:0/2798793282 conn(0x7f64040a45d0 msgr2=0x7f64040a36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 shutdown_connections 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 -- 172.21.15.5:0/2798793282 wait complete. 2024-09-16T10:56:10.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.418+0000 7f640359e700 1 librados: shutdown 2024-09-16T10:56:10.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:56:10.455 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T10:56:10.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:56:10.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 10 monclient: get_monmap_and_config 2024-09-16T10:56:10.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 10 monclient: build_initial_monmap 2024-09-16T10:56:10.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 10 monclient: monmap: 2024-09-16T10:56:10.576 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:56:10.576050+0000 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:56:10.576050+0000 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:56:10.577 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 1 Processor -- start 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 1 -- start start 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.570+0000 7f3b009e9700 10 monclient: init 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-b con 0x7f3afc09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3afc09d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 0x7f3afc09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-c con 0x7f3afc09dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3afc070e40 con 0x7f3afc09d6a0 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc0a01d0 con 0x7f3afc09dfd0 2024-09-16T10:56:10.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc09f740 con 0x7f3afc09e8e0 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient(hunting): authenticate will time out at 1229.152451s 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afbfff700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 msgr2=0x7f3afc09f200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 0x7f3afc09f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3af3fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3afc09d6a0 0x7f3afc1336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:32798/0 (socket says 172.21.15.5:32798) 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3af3fff700 1 -- 172.21.15.5:0/1072911280 learned_addr learned my addr 172.21.15.5:0/1072911280 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): get_auth_request con 0x7f3afc09e8e0 auth_method 0 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_done global_id 54200 payload 274 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 1 -- 172.21.15.5:0/1072911280 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196955107 0 0) 0x7f3afc070e40 con 0x7f3afc09d6a0 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 -- 172.21.15.5:0/1072911280 >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 msgr2=0x7f3afc09f200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 --2- 172.21.15.5:0/1072911280 >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 0x7f3afc09f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 -- 172.21.15.5:0/1072911280 >> v1:172.21.15.5:6789/0 conn(0x7f3afc09d6a0 legacy=0x7f3afc1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 monclient: found mon.noname-b 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:56:10.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 -- 172.21.15.5:0/1072911280 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc09fef0 con 0x7f3afc09e8e0 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afb7fe700 1 --2- 172.21.15.5:0/1072911280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc09ecc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009850 tx=0x7f3aec009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d75092967b0575 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 1 -- 172.21.15.5:0/1072911280 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec011830 con 0x7f3afc09e8e0 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: dump: 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.590 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 5 monclient: authenticate success, global_id 54200 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 1 -- 172.21.15.5:0/1072911280 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec011990 con 0x7f3afc09e8e0 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient: get_monmap_and_config success 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 4 set_mon_vals no callback set 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 1 -- 172.21.15.5:0/1072911280 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec011d20 con 0x7f3afc09e8e0 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3afaffd700 10 monclient: dump: 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 10 monclient: shutdown 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- 172.21.15.5:0/1072911280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 msgr2=0x7f3afc09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 --2- 172.21.15.5:0/1072911280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc09ecc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3aec009850 tx=0x7f3aec009b60 comp rx=0 tx=0).stop 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- 172.21.15.5:0/1072911280 shutdown_connections 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 --2- 172.21.15.5:0/1072911280 >> v2:172.21.15.143:3300/0 conn(0x7f3afc09dfd0 0x7f3afc09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 --2- 172.21.15.5:0/1072911280 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc09ecc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- 172.21.15.5:0/1072911280 >> 172.21.15.5:0/1072911280 conn(0x7f3afc0a45d0 msgr2=0x7f3afc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- 172.21.15.5:0/1072911280 shutdown_connections 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.574+0000 7f3b009e9700 1 -- 172.21.15.5:0/1072911280 wait complete. 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: build_initial_monmap 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: monmap: 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:56:10.593 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 librados: starting msgr at 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 librados: starting objecter 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 Processor -- start 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- start start 2024-09-16T10:56:10.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 librados: setting wanted keys 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 librados: calling monclient init 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: init 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _reopen_session rank -1 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-a con 0x7f3afc09d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc09dfd0 0x7f3afc1cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-b con 0x7f3afc09dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc122230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): picked mon.noname-c con 0x7f3afc09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc124960 con 0x7f3afc09d6a0 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc124790 con 0x7f3afc09dfd0 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc09b0d0 con 0x7f3afc09e8e0 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): _renew_subs 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): authenticate will time out at 1229.157319s 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51364/0 (socket says 172.21.15.5:51364) 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 -- 172.21.15.5:0/213517788 learned_addr learned my addr 172.21.15.5:0/213517788 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): get_auth_request con 0x7f3afc09d6a0 auth_method 0 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc122230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af3fff700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc09dfd0 msgr2=0x7f3afc1cb130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af3fff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc09dfd0 0x7f3afc1cb130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 10 monclient(hunting): get_auth_request con 0x7f3afc09e8e0 auth_method 0 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient(hunting): handle_auth_done global_id 28806 payload 824 2024-09-16T10:56:10.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 msgr2=0x7f3afc122230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc122230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc09dfd0 msgr2=0x7f3afc1cb130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3afc09dfd0 0x7f3afc1cb130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 monclient: found mon.noname-a 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0094b0 con 0x7f3afc09d6a0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc09e8e0 0x7f3afc122230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 client.?.objecter ms_handle_connect 0x7f3afc09d6a0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afb7fe700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 secure :-1 s=READY pgs=6873 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0a0310 tx=0x7f3aec004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0038c0 con 0x7f3afc09d6a0 2024-09-16T10:56:10.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: dump: 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: _finish_auth 0 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 5 monclient: authenticate success, global_id 28806 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec01f070 con 0x7f3afc09d6a0 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _renew_subs 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc126160 con 0x7f3afc09d6a0 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003c60 con 0x7f3afc09d6a0 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 monclient: dump: 2024-09-16T10:56:10.600 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3afa7fc700 4 set_mon_vals no callback set 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 client.28806.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _renew_subs 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:56:10.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc0a3900 con 0x7f3afc09d6a0 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 librados: init done 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3aec01a430 con 0x7f3afc09d6a0 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 --2- 172.21.15.5:0/213517788 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3adc036f10 0x7f3adc0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 0x7f3afc122b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc067380 con 0x7f3afc122770 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 6092+0+0 (secure 0 0 0) 0x7f3aec021070 con 0x7f3afc09d6a0 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 10 client.28806.objecter ms_dispatch 0x7f3afc000fd0 osd_map(276..276 src has 1..276) v4 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 3 client.28806.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.578+0000 7f3af8ff9700 3 client.28806.objecter handle_osd_map decoding full epoch 276 2024-09-16T10:56:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 20 client.28806.objecter dump_active .. 0 homeless 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 0x7f3afc122b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af3fff700 1 --2- 172.21.15.5:0/213517788 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3adc036f10 0x7f3adc0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient: get_auth_request con 0x7f3afc122770 auth_method 0 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af3fff700 10 monclient: get_auth_request con 0x7f3adc036f10 auth_method 0 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 10 monclient(hunting): handle_auth_done global_id 54206 payload 824 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3afc04ecd0 con 0x7f3afc122770 2024-09-16T10:56:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 client.28806.objecter ms_handle_connect 0x7f3adc036f10 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af3fff700 1 --2- 172.21.15.5:0/213517788 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3adc036f10 0x7f3adc0393d0 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f3afc126370 tx=0x7f3ae4007b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 client.28806.objecter ms_handle_connect 0x7f3afc122770 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 client.28806.objecter resend_mon_ops 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 0x7f3afc122b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3afc06b9c0 tx=0x7f3af400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4015630 con 0x7f3afc122770 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 monclient: dump: 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.605 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f3af4015790 con 0x7f3afc122770 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 msgr2=0x7f3afc122b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 0x7f3afc122b50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3afc06b9c0 tx=0x7f3af400b690 comp rx=0 tx=0).stop 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 msgr2=0x7f3afc122b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.582+0000 7f3af8ff9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc122770 0x7f3afc122b50 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3afc06b9c0 tx=0x7f3af400b690 comp rx=0 tx=0).stop 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3b009e9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3b009e9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 0x7f3afc11e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3b009e9700 10 monclient(hunting): start opening mon connection 2024-09-16T10:56:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc1341b0 con 0x7f3afc123090 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 0x7f3afc11e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3afbfff700 10 monclient: get_auth_request con 0x7f3afc123090 auth_method 0 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3afbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3afbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.590+0000 7f3afbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:56:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 10 monclient(hunting): handle_auth_done global_id 54209 payload 824 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 1 -- 172.21.15.5:0/213517788 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3afc123d90 con 0x7f3afc123090 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 0x7f3afc11e940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009fd0 tx=0x7f3af4003c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 client.28806.objecter ms_handle_connect 0x7f3afc123090 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 client.28806.objecter resend_mon_ops 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4022070 con 0x7f3afc123090 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 monclient: dump: 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.609 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7f3af401dd70 con 0x7f3afc123090 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 msgr2=0x7f3afc11e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 0x7f3afc11e940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009fd0 tx=0x7f3af4003c90 comp rx=0 tx=0).stop 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 msgr2=0x7f3afc11e940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:56:10.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3af8ff9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3afc123090 0x7f3afc11e940 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009fd0 tx=0x7f3af4003c90 comp rx=0 tx=0).stop 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 34, 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.612 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.613 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:56:10.614 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:56:10.615 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.616 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.617 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.619 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:56:10.620 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:56:10.621 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.622 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:56:10.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:56:10.624 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:56:10.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:56:10.624 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:56:10.624 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:56:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 10 librados: watch_flush enter 2024-09-16T10:56:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 10 librados: watch_flush exit 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 20 client.28806.objecter shutdown clearing up homeless session... 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 10 client.28806.objecter successfully canceled tick 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3adc036f10 msgr2=0x7f3adc0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 --2- 172.21.15.5:0/213517788 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3adc036f10 0x7f3adc0393d0 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f3afc126370 tx=0x7f3ae4007b70 comp rx=0 tx=0).stop 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 10 monclient: shutdown 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 msgr2=0x7f3afc1cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3afbfff700 1 -- 172.21.15.5:0/213517788 reap_dead start 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 secure :-1 s=READY pgs=6873 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0a0310 tx=0x7f3aec004770 comp rx=0 tx=0).stop 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 shutdown_connections 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 --2- 172.21.15.5:0/213517788 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3afc09d6a0 0x7f3afc1cabf0 unknown :-1 s=CLOSED pgs=6873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 >> 172.21.15.5:0/213517788 conn(0x7f3afc0a45d0 msgr2=0x7f3afc09cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:56:10.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 shutdown_connections 2024-09-16T10:56:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 -- 172.21.15.5:0/213517788 wait complete. 2024-09-16T10:56:10.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:56:10.594+0000 7f3b009e9700 1 librados: shutdown 2024-09-16T10:56:10.627 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:56:10.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout:63:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:5d165639:::164:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f43765fc:::165:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b4c720e9:::166:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e694b040:::167:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:afa38db2:::168:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:77ba9f53:::169:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:87495034:::170:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7c96bf0e:::171:head 2024-09-16T10:56:12.885 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:dbe346cc:::172:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:e943ec24:::173:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f97a9c0c:::174:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:6f26e74d:::175:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4f95e106:::176:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:0e6f2f8f:::177:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:05db05f1:::178:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:38a78d66:::179:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:d095610b:::180:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:a1a9d709:::181:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:1e5d39db:::182:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:f7df4fb9:::183:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:03a7f161:::184:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ba70721e:::185:head 2024-09-16T10:56:12.886 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:28e5662d:::186:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:973d52de:::187:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:4303eb1c:::188:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:b990b48e:::189:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:29b8165b:::190:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:3547f197:::191:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:7e260936:::192:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:1abec7b1:::193:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:10fdda93:::194:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:15817eea:::195:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:770bab57:::196:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:ed9e13e7:::197:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:71471a8f:::198:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: checking for 190:10fb1d02:::199:head 2024-09-16T10:56:12.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8923 ms) 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,0 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first is 1726484118 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,0 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,0 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,0 2024-09-16T10:56:12.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,1726484126,1726484127,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,1726484126,1726484127,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484118,1726484120,1726484121,1726484123,1726484124,1726484126,1726484127,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484121,1726484123,1726484124,1726484126,1726484127,1726484129,1726484130,0 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first now 1726484121, trimmed 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21022 ms) 2024-09-16T10:56:12.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T10:56:12.890 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: foo0 2024-09-16T10:56:12.890 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T10:56:12.890 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14026 ms) 2024-09-16T10:56:12.890 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T10:56:40.794 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3018 ms) 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6994 ms) 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (21112 ms) 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6885 ms) 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T10:56:40.795 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7040 ms) 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6998 ms) 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6977 ms) 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (167937 ms total) 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (332658 ms total) 2024-09-16T10:56:40.796 INFO:tasks.workunit.client.0.smithi143.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T10:57:40.628 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T10:57:40.629 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T10:57:40.629 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:57:40.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:57:40.633 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:57:40.633 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:57:40.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:40.752 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T10:57:40.745+0000 7f4ea6a60d40 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 273 -> 0 (sync_force? mon rebuild?) 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: get_monmap_and_config 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: build_initial_monmap 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: monmap: 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:40.758404+0000 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:40.758404+0000 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:40.759 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:40.760 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 Processor -- start 2024-09-16T10:57:40.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 -- start start 2024-09-16T10:57:40.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: init 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f321c0a2eb0 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 0x7f321c0a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): picked mon.noname-b con 0x7f321c0a4280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f321c13aeb0 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c09fdc0 con 0x7f321c13aeb0 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c09fae0 con 0x7f321c0a2eb0 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0a1e50 con 0x7f321c0a4280 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:40.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f3221385700 10 monclient(hunting): authenticate will time out at 1319.335584s 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321b7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f321c13aeb0 0x7f321c13d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58500/0 (socket says 172.21.15.5:58500) 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321a7fc700 1 --2- 172.21.15.5:0/2010192012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 0x7f321c0a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321b7fe700 1 -- 172.21.15.5:0/2010192012 learned_addr learned my addr 172.21.15.5:0/2010192012 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321a7fc700 10 monclient(hunting): get_auth_request con 0x7f321c0a4280 auth_method 0 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.756+0000 7f321a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:40.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient(hunting): handle_auth_done global_id 54230 payload 274 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 1 -- 172.21.15.5:0/2010192012 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494184240 0 0) 0x7f321c09fdc0 con 0x7f321c13aeb0 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 1 -- 172.21.15.5:0/2010192012 >> v1:172.21.15.143:6789/0 conn(0x7f321c0a2eb0 legacy=0x7f321c0a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 1 -- 172.21.15.5:0/2010192012 >> v1:172.21.15.5:6789/0 conn(0x7f321c13aeb0 legacy=0x7f321c13d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:40.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient: _finish_hunting 0 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 1 monclient: found mon.noname-b 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 1 -- 172.21.15.5:0/2010192012 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c09efb0 con 0x7f321c0a4280 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f321a7fc700 1 --2- 172.21.15.5:0/2010192012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 0x7f321c0a39a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3204009880 tx=0x7f3204009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c6ed0660cae26f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 1 -- 172.21.15.5:0/2010192012 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204011830 con 0x7f321c0a4280 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: dump: 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:40.778 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: _finish_auth 0 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 1 -- 172.21.15.5:0/2010192012 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3204011990 con 0x7f321c0a4280 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 5 monclient: authenticate success, global_id 54230 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:40.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 4 set_mon_vals no callback set 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 1 -- 172.21.15.5:0/2010192012 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204011d20 con 0x7f321c0a4280 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3219ffb700 10 monclient: dump: 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:40.780 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 10 monclient: shutdown 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- 172.21.15.5:0/2010192012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 msgr2=0x7f321c0a39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 --2- 172.21.15.5:0/2010192012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 0x7f321c0a39a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3204009880 tx=0x7f3204009b90 comp rx=0 tx=0).stop 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- 172.21.15.5:0/2010192012 shutdown_connections 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 --2- 172.21.15.5:0/2010192012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c0a4280 0x7f321c0a39a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- 172.21.15.5:0/2010192012 >> 172.21.15.5:0/2010192012 conn(0x7f321c0a4fb0 msgr2=0x7f321c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- 172.21.15.5:0/2010192012 shutdown_connections 2024-09-16T10:57:40.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- 172.21.15.5:0/2010192012 wait complete. 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 10 monclient: build_initial_monmap 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 10 monclient: monmap: 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 librados: starting msgr at 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 librados: starting objecter 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 Processor -- start 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 -- start start 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 librados: setting wanted keys 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 1 librados: calling monclient init 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.760+0000 7f3221385700 10 monclient: init 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): picked mon.noname-b con 0x7f321c0a2eb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f321c0a4280 0x7f321c11f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): picked mon.noname-a con 0x7f321c0a4280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:40.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): picked mon.noname-c con 0x7f321c13aeb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c121fb0 con 0x7f321c0a4280 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c1236b0 con 0x7f321c0a2eb0 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0a72d0 con 0x7f321c13aeb0 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient(hunting): authenticate will time out at 1319.340332s 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f321c0a4280 0x7f321c11f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:58116/0 (socket says 172.21.15.5:58116) 2024-09-16T10:57:40.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 -- 172.21.15.5:0/1362205003 learned_addr learned my addr 172.21.15.5:0/1362205003 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): get_auth_request con 0x7f321c13aeb0 auth_method 0 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): get_auth_request con 0x7f321c0a4280 auth_method 0 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 10 monclient(hunting): get_auth_request con 0x7f321c0a2eb0 auth_method 0 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:40.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:40.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 -- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 msgr2=0x7f321c139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 -- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 msgr2=0x7f321c139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient(hunting): handle_auth_done global_id 54233 payload 824 2024-09-16T10:57:40.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 -- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 msgr2=0x7f321c139bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 -- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f321c0a4280 msgr2=0x7f321c11f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f321c0a4280 0x7f321c11f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 monclient: found mon.noname-c 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 -- 172.21.15.5:0/1362205003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040094b0 con 0x7f321c13aeb0 2024-09-16T10:57:40.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321b7fe700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f320c00c430 tx=0x7f320c00c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 client.?.objecter ms_handle_connect 0x7f321c13aeb0 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c004ca0 con 0x7f321c13aeb0 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: dump: 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:40.789 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: _finish_auth 0 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: _check_auth_tickets 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320c004e20 con 0x7f321c13aeb0 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:40.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c007310 con 0x7f321c13aeb0 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 5 monclient: authenticate success, global_id 54233 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _renew_subs 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3218ff9700 4 set_mon_vals no callback set 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c13e5d0 con 0x7f321c13aeb0 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 monclient: dump: 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.791 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 client.54233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _renew_subs 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c1233d0 con 0x7f321c13aeb0 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 librados: init done 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f320c007470 con 0x7f321c13aeb0 2024-09-16T10:57:40.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 --2- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 0x7f31fc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(339..339 src has 57..339) v4 ==== 5373+0+0 (secure 0 0 0) 0x7f320c047a90 con 0x7f321c13aeb0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 10 client.54233.objecter ms_dispatch 0x7f321c000fd0 osd_map(339..339 src has 57..339) v4 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 3 client.54233.objecter handle_osd_map got epochs [339,339] > 0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 3 client.54233.objecter handle_osd_map decoding full epoch 339 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c0a3290 con 0x7f321c13aeb0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f32137fe700 20 client.54233.objecter dump_active .. 0 homeless 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.764+0000 7f321affd700 1 --2- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 0x7f31fc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.768+0000 7f321affd700 10 monclient: get_auth_request con 0x7f31fc036f60 auth_method 0 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.768+0000 7f32137fe700 10 client.54233.objecter ms_handle_connect 0x7f31fc036f60 2024-09-16T10:57:40.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.768+0000 7f321affd700 1 --2- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 0x7f31fc039420 secure :-1 s=READY pgs=3797 cs=0 l=1 rev1=1 crypto rx=0x7f32140012e0 tx=0x7f3214009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:40.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.772+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f320c007720 con 0x7f321c13aeb0 2024-09-16T10:57:40.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.772+0000 7f32137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:40.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:40.772+0000 7f32137fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:41.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.012+0000 7f3221385700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:41.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.012+0000 7f3221385700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:41.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.012+0000 7f3221385700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:41.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.012+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f321c072380 con 0x7f321c13aeb0 2024-09-16T10:57:41.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f32137fe700 1 -- 172.21.15.5:0/1362205003 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f320c034070 con 0x7f321c13aeb0 2024-09-16T10:57:41.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f32137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:41.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f32137fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:41.018 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 10 librados: watch_flush enter 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 10 librados: watch_flush exit 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 20 client.54233.objecter shutdown clearing up homeless session... 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 10 client.54233.objecter successfully canceled tick 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 msgr2=0x7f31fc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 0x7f31fc039420 secure :-1 s=READY pgs=3797 cs=0 l=1 rev1=1 crypto rx=0x7f32140012e0 tx=0x7f3214009040 comp rx=0 tx=0).stop 2024-09-16T10:57:41.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 10 monclient: shutdown 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 msgr2=0x7f321c11fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f320c00c430 tx=0x7f320c00c740 comp rx=0 tx=0).stop 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 shutdown_connections 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f321c13aeb0 0x7f321c11fbf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f31fc036f60 0x7f31fc039420 unknown :-1 s=CLOSED pgs=3797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f321c0a4280 0x7f321c11f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 --2- 172.21.15.5:0/1362205003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f321c0a2eb0 0x7f321c139bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 >> 172.21.15.5:0/1362205003 conn(0x7f321c0a4fb0 msgr2=0x7f321c09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 shutdown_connections 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 -- 172.21.15.5:0/1362205003 wait complete. 2024-09-16T10:57:41.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:41.016+0000 7f3221385700 1 librados: shutdown 2024-09-16T10:57:44.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.152+0000 7fdc4dcea700 10 monclient: get_monmap_and_config 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.152+0000 7fdc4dcea700 10 monclient: build_initial_monmap 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.152+0000 7fdc4dcea700 10 monclient: monmap: 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:44.156547+0000 2024-09-16T10:57:44.157 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:44.156547+0000 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:44.158 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 Processor -- start 2024-09-16T10:57:44.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- start start 2024-09-16T10:57:44.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient: init 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdc4809e8e0 0x7fdc4809ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-a con 0x7fdc4809e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-b con 0x7fdc4809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc4809dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc48070e40 con 0x7fdc4809dfd0 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc480a01d0 con 0x7fdc4809e8e0 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:44.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4809f740 con 0x7fdc4809d6a0 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient(hunting): authenticate will time out at 1322.733408s 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdc4809e8e0 0x7fdc4809ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48574/0 (socket says 172.21.15.5:48574) 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 -- 172.21.15.5:0/1058077837 learned_addr learned my addr 172.21.15.5:0/1058077837 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): get_auth_request con 0x7fdc4809d6a0 auth_method 0 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): get_auth_request con 0x7fdc4809e8e0 auth_method 0 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:44.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4cce8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient(hunting): handle_auth_done global_id 54236 payload 274 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 -- 172.21.15.5:0/1058077837 >> v2:172.21.15.5:3300/0 conn(0x7fdc4809e8e0 msgr2=0x7fdc4809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 --2- 172.21.15.5:0/1058077837 >> v2:172.21.15.5:3300/0 conn(0x7fdc4809e8e0 0x7fdc4809ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 -- 172.21.15.5:0/1058077837 >> v1:172.21.15.143:6789/0 conn(0x7fdc4809dfd0 legacy=0x7fdc481336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:44.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 monclient: found mon.noname-b 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 -- 172.21.15.5:0/1058077837 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4809fef0 con 0x7fdc4809d6a0 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc47fff700 1 --2- 172.21.15.5:0/1058077837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e3723b1d335be48 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 1 -- 172.21.15.5:0/1058077837 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4000e070 con 0x7fdc4809d6a0 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: dump: 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.166 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: _finish_auth 0 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: _check_auth_tickets 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 5 monclient: authenticate success, global_id 54236 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 1 -- 172.21.15.5:0/1058077837 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4000b910 con 0x7fdc4809d6a0 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:44.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 4 set_mon_vals no callback set 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 1 -- 172.21.15.5:0/1058077837 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4000bcc0 con 0x7fdc4809d6a0 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc477fe700 10 monclient: dump: 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:44.168 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 10 monclient: shutdown 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- 172.21.15.5:0/1058077837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 msgr2=0x7fdc4809f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/1058077837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4000b330 comp rx=0 tx=0).stop 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- 172.21.15.5:0/1058077837 shutdown_connections 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/1058077837 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809d6a0 0x7fdc4809f200 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/1058077837 >> v2:172.21.15.5:3300/0 conn(0x7fdc4809e8e0 0x7fdc4809ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- 172.21.15.5:0/1058077837 >> 172.21.15.5:0/1058077837 conn(0x7fdc480a45d0 msgr2=0x7fdc480a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- 172.21.15.5:0/1058077837 shutdown_connections 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.156+0000 7fdc4dcea700 1 -- 172.21.15.5:0/1058077837 wait complete. 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient: build_initial_monmap 2024-09-16T10:57:44.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient: monmap: 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:44.170 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 librados: starting msgr at 2024-09-16T10:57:44.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 librados: starting objecter 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 Processor -- start 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 -- start start 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 librados: setting wanted keys 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 librados: calling monclient init 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient: init 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-b con 0x7fdc4809d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-a con 0x7fdc4809dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809e8e0 0x7fdc4811f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): picked mon.noname-c con 0x7fdc4809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48121420 con 0x7fdc4809dfd0 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480a3a20 con 0x7fdc4809d6a0 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48133750 con 0x7fdc4809e8e0 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4dcea700 10 monclient(hunting): authenticate will time out at 1322.739670s 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4d4e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809e8e0 0x7fdc4811f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50934/0 (socket says 172.21.15.5:50934) 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 1 -- 172.21.15.5:0/2385724768 learned_addr learned my addr 172.21.15.5:0/2385724768 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:44.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 10 monclient(hunting): get_auth_request con 0x7fdc4809dfd0 auth_method 0 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4d4e9700 10 monclient(hunting): get_auth_request con 0x7fdc4809e8e0 auth_method 0 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4d4e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4d4e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.160+0000 7fdc4d4e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 10 monclient(hunting): get_auth_request con 0x7fdc4809d6a0 auth_method 0 2024-09-16T10:57:44.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4d4e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4d4e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4d4e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 1 -- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 msgr2=0x7fdc4811eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 1 -- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 msgr2=0x7fdc4811eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4cce8700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient(hunting): handle_auth_done global_id 28842 payload 824 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 -- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809e8e0 msgr2=0x7fdc4811f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809e8e0 0x7fdc4811f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 -- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 msgr2=0x7fdc4811eeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 monclient: found mon.noname-a 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 -- 172.21.15.5:0/2385724768 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40009b30 con 0x7fdc4809dfd0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc47fff700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 secure :-1 s=READY pgs=6887 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b370 tx=0x7fdc40004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 client.?.objecter ms_handle_connect 0x7fdc4809dfd0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc400046a0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:44.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: dump: 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: _finish_auth 0 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: _check_auth_tickets 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:44.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc400092e0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 5 monclient: authenticate success, global_id 28842 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4001c440 con 0x7fdc4809dfd0 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc46ffd700 4 set_mon_vals no callback set 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _renew_subs 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4809b0d0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 monclient: dump: 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:44.182 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 client.28842.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _renew_subs 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48122bb0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 1 librados: init done 2024-09-16T10:57:44.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fdc4001c5a0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc481341b0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 --2- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 0x7fdc2c039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(340..340 src has 57..340) v4 ==== 5345+0+0 (secure 0 0 0) 0x7fdc40049640 con 0x7fdc4809dfd0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 10 client.28842.objecter ms_dispatch 0x7fdc48000fd0 osd_map(340..340 src has 57..340) v4 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 3 client.28842.objecter handle_osd_map got epochs [340,340] > 0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 3 client.28842.objecter handle_osd_map decoding full epoch 340 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.164+0000 7fdc457fa700 20 client.28842.objecter dump_active .. 0 homeless 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.168+0000 7fdc4cce8700 1 --2- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 0x7fdc2c039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.168+0000 7fdc4cce8700 10 monclient: get_auth_request con 0x7fdc2c036f90 auth_method 0 2024-09-16T10:57:44.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.168+0000 7fdc457fa700 10 client.28842.objecter ms_handle_connect 0x7fdc2c036f90 2024-09-16T10:57:44.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.168+0000 7fdc4cce8700 1 --2- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 0x7fdc2c039450 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7fdc300095b0 tx=0x7fdc30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:44.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.176+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdc400249a0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.176+0000 7fdc457fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:44.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.176+0000 7fdc457fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc4dcea700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc4dcea700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc4dcea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:44.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc480008d0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc457fa700 1 -- 172.21.15.5:0/2385724768 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fdc400243b0 con 0x7fdc4809dfd0 2024-09-16T10:57:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc457fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:44.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.404+0000 7fdc457fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:44.409 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 10 librados: watch_flush enter 2024-09-16T10:57:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 10 librados: watch_flush exit 2024-09-16T10:57:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 20 client.28842.objecter shutdown clearing up homeless session... 2024-09-16T10:57:44.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 10 client.28842.objecter successfully canceled tick 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 msgr2=0x7fdc2c039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 0x7fdc2c039450 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7fdc300095b0 tx=0x7fdc30008040 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 10 monclient: shutdown 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 msgr2=0x7fdc4811f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 secure :-1 s=READY pgs=6887 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b370 tx=0x7fdc40004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 shutdown_connections 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdc4809e8e0 0x7fdc4811f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdc2c036f90 0x7fdc2c039450 unknown :-1 s=CLOSED pgs=3798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdc4809dfd0 0x7fdc4811f3f0 unknown :-1 s=CLOSED pgs=6887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 --2- 172.21.15.5:0/2385724768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc4809d6a0 0x7fdc4811eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 >> 172.21.15.5:0/2385724768 conn(0x7fdc480a45d0 msgr2=0x7fdc48132fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:44.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 shutdown_connections 2024-09-16T10:57:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 -- 172.21.15.5:0/2385724768 wait complete. 2024-09-16T10:57:44.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:44.408+0000 7fdc4dcea700 1 librados: shutdown 2024-09-16T10:57:47.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:47.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.540+0000 7ff113fff700 10 monclient: get_monmap_and_config 2024-09-16T10:57:47.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.540+0000 7ff113fff700 10 monclient: build_initial_monmap 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.540+0000 7ff113fff700 10 monclient: monmap: 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:47.543911+0000 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:47.543911+0000 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:47.545 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:47.546 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:47.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 Processor -- start 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- start start 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient: init 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1140a0970 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 0x7ff1140a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff1140a0d50 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 0x7ff11413d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff11413ac10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:47.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff114099e80 con 0x7ff1140a0970 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11409a790 con 0x7ff1140a0d50 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11409a440 con 0x7ff11413ac10 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient(hunting): authenticate will time out at 1326.121013s 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 0x7ff1140a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff112ffd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff1140a0970 0x7ff11409ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43850/0 (socket says 172.21.15.5:43850) 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff112ffd700 1 -- 172.21.15.5:0/1455117847 learned_addr learned my addr 172.21.15.5:0/1455117847 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 --2- 172.21.15.5:0/1455117847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 0x7ff11413d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): get_auth_request con 0x7ff1140a0d50 auth_method 0 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:47.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): get_auth_request con 0x7ff11413ac10 auth_method 0 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient(hunting): handle_auth_done global_id 54242 payload 274 2024-09-16T10:57:47.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 -- 172.21.15.5:0/1455117847 >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 msgr2=0x7ff1140a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 --2- 172.21.15.5:0/1455117847 >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 0x7ff1140a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 -- 172.21.15.5:0/1455117847 >> v1:172.21.15.5:6789/0 conn(0x7ff1140a0970 legacy=0x7ff11409ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 monclient: found mon.noname-b 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 -- 172.21.15.5:0/1455117847 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11409a160 con 0x7ff11413ac10 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff10a7fc700 1 --2- 172.21.15.5:0/1455117847 >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 0x7ff1140a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1137fe700 1 --2- 172.21.15.5:0/1455117847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 0x7ff11413d000 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009d90 tx=0x7ff10c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c096c0f4d3217d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 1 -- 172.21.15.5:0/1455117847 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00e070 con 0x7ff11413ac10 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:47.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: dump: 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: _finish_auth 0 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: _check_auth_tickets 2024-09-16T10:57:47.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 1 -- 172.21.15.5:0/1455117847 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c00bb20 con 0x7ff11413ac10 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 5 monclient: authenticate success, global_id 54242 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 1 -- 172.21.15.5:0/1455117847 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c012430 con 0x7ff11413ac10 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 4 set_mon_vals no callback set 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff1127fc700 10 monclient: dump: 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:47.555 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 10 monclient: shutdown 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- 172.21.15.5:0/1455117847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 msgr2=0x7ff11413d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 --2- 172.21.15.5:0/1455117847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 0x7ff11413d000 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009d90 tx=0x7ff10c00b330 comp rx=0 tx=0).stop 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- 172.21.15.5:0/1455117847 shutdown_connections 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 --2- 172.21.15.5:0/1455117847 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff11413ac10 0x7ff11413d000 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 --2- 172.21.15.5:0/1455117847 >> v2:172.21.15.5:3300/0 conn(0x7ff1140a0d50 0x7ff1140a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- 172.21.15.5:0/1455117847 >> 172.21.15.5:0/1455117847 conn(0x7ff1140a2090 msgr2=0x7ff11406a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- 172.21.15.5:0/1455117847 shutdown_connections 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.544+0000 7ff113fff700 1 -- 172.21.15.5:0/1455117847 wait complete. 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient: build_initial_monmap 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient: monmap: 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:47.557 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:47.558 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 librados: starting msgr at 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 librados: starting objecter 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 Processor -- start 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 -- start start 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 librados: setting wanted keys 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 librados: calling monclient init 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient: init 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1140a0970 0x7ff1141357f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff1140a0970 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff1140a0d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff11413ac10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11409c150 con 0x7ff11413ac10 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11413b180 con 0x7ff1140a0d50 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11413d150 con 0x7ff1140a0970 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff113fff700 10 monclient(hunting): authenticate will time out at 1326.126661s 2024-09-16T10:57:47.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1140a0970 0x7ff1141357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50956/0 (socket says 172.21.15.5:50956) 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 -- 172.21.15.5:0/601506384 learned_addr learned my addr 172.21.15.5:0/601506384 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): get_auth_request con 0x7ff11413ac10 auth_method 0 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): get_auth_request con 0x7ff1140a0970 auth_method 0 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:47.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff112ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 10 monclient(hunting): get_auth_request con 0x7ff1140a0d50 auth_method 0 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient(hunting): handle_auth_done global_id 28851 payload 824 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 1 -- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 msgr2=0x7ff114135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:57:47.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 -- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1140a0970 msgr2=0x7ff1141357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 1 -- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 msgr2=0x7ff114135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1140a0970 0x7ff1141357f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 -- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 msgr2=0x7ff114135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff10a7fc700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 monclient: found mon.noname-a 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.548+0000 7ff1137fe700 1 -- 172.21.15.5:0/601506384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc0094b0 con 0x7ff11413ac10 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff1137fe700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 secure :-1 s=READY pgs=6889 cs=0 l=1 rev1=1 crypto rx=0x7ff10c014040 tx=0x7ff10c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:47.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 client.?.objecter ms_handle_connect 0x7ff11413ac10 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c012430 con 0x7ff11413ac10 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: dump: 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:47.568 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: _finish_auth 0 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 5 monclient: authenticate success, global_id 28851 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c020070 con 0x7ff11413ac10 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c004040 con 0x7ff11413ac10 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _renew_subs 2024-09-16T10:57:47.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff111ffb700 4 set_mon_vals no callback set 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c009a70 con 0x7ff11413ac10 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 monclient: dump: 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:47.570 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 client.28851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _renew_subs 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141202c0 con 0x7ff11413ac10 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 1 librados: init done 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1140a11c0 con 0x7ff11413ac10 2024-09-16T10:57:47.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff10c0041a0 con 0x7ff11413ac10 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 --2- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 0x7ff0f4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(340..340 src has 57..340) v4 ==== 5345+0+0 (secure 0 0 0) 0x7ff10c022070 con 0x7ff11413ac10 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 client.28851.objecter ms_dispatch 0x7ff114000fd0 osd_map(340..340 src has 57..340) v4 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 3 client.28851.objecter handle_osd_map got epochs [340,340] > 0 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 3 client.28851.objecter handle_osd_map decoding full epoch 340 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 20 client.28851.objecter dump_active .. 0 homeless 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff112ffd700 1 --2- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 0x7ff0f4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff112ffd700 10 monclient: get_auth_request con 0x7ff0f4036f60 auth_method 0 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff10bfff700 10 client.28851.objecter ms_handle_connect 0x7ff0f4036f60 2024-09-16T10:57:47.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.552+0000 7ff112ffd700 1 --2- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 0x7ff0f4039420 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7ff104005930 tx=0x7ff104009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.560+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff10c01a970 con 0x7ff11413ac10 2024-09-16T10:57:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.560+0000 7ff10bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:47.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.560+0000 7ff10bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:47.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff113fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:47.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff113fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:47.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff113fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:47.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff11411e8b0 con 0x7ff11413ac10 2024-09-16T10:57:47.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff10bfff700 1 -- 172.21.15.5:0/601506384 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7ff10c01a380 con 0x7ff11413ac10 2024-09-16T10:57:47.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff10bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:47.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.792+0000 7ff10bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:47.796 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 10 librados: watch_flush enter 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 10 librados: watch_flush exit 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 20 client.28851.objecter shutdown clearing up homeless session... 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 10 client.28851.objecter successfully canceled tick 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 msgr2=0x7ff0f4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 0x7ff0f4039420 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7ff104005930 tx=0x7ff104009040 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 10 monclient: shutdown 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 msgr2=0x7ff11411c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 secure :-1 s=READY pgs=6889 cs=0 l=1 rev1=1 crypto rx=0x7ff10c014040 tx=0x7ff10c004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 shutdown_connections 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0f4036f60 0x7ff0f4039420 unknown :-1 s=CLOSED pgs=3799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff11413ac10 0x7ff11411c360 unknown :-1 s=CLOSED pgs=6889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1140a0d50 0x7ff114135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 --2- 172.21.15.5:0/601506384 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff1140a0970 0x7ff1141357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 >> 172.21.15.5:0/601506384 conn(0x7ff1140a2090 msgr2=0x7ff1140a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 shutdown_connections 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 -- 172.21.15.5:0/601506384 wait complete. 2024-09-16T10:57:47.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:47.796+0000 7ff113fff700 1 librados: shutdown 2024-09-16T10:57:50.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: get_monmap_and_config 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: build_initial_monmap 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: monmap: 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.930 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:50.929587+0000 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:50.929587+0000 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:50.931 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 Processor -- start 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 -- start start 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: init 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 0x7f0294023ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-c con 0x7f0294029580 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02940248c0 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02940bbca0 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:50.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029402a390 con 0x7f02940bbca0 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294029960 con 0x7f02940248c0 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02940bc6e0 con 0x7f0294029580 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a2184700 10 monclient(hunting): authenticate will time out at 1329.506596s 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1983700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f02940bbca0 0x7f02940256d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43862/0 (socket says 172.21.15.5:43862) 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1983700 1 -- 172.21.15.5:0/3485312718 learned_addr learned my addr 172.21.15.5:0/3485312718 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 1 --2- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 0x7f0294023ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 10 monclient(hunting): get_auth_request con 0x7f0294029580 auth_method 0 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 1 -- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 msgr2=0x7f0294023ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 1 -- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 msgr2=0x7f0294023ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 1 --2- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 0x7f0294023ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f02a1182700 1 --2- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 0x7f0294023ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736288566 0 0) 0x7f029402a390 con 0x7f02940bbca0 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 10 monclient(hunting): _negotiate 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 10 monclient(hunting): my global_id is 28854 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.928+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02880035d0 con 0x7f02940bbca0 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449882542 0 0) 0x7f02880035d0 con 0x7f02940bbca0 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 msgr2=0x7f0294023ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:50.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 --2- 172.21.15.5:0/3485312718 >> v2:172.21.15.143:3300/0 conn(0x7f0294029580 0x7f0294023ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 >> v1:172.21.15.143:6789/0 conn(0x7f02940248c0 legacy=0x7f0294024fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294025de0 con 0x7f02940bbca0 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _finish_auth 0 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _check_auth_tickets 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 5 monclient: authenticate success, global_id 28854 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f029c067140 con 0x7f02940bbca0 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:50.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: dump: 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 >> v1:172.21.15.5:6789/0 conn(0x7f02940bbca0 legacy=0x7f02940256d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0288003a20 0x7f0288003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): picked mon.c con 0x7f0288003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 0x7f0288004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): picked mon.b con 0x7f0288004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f028800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): picked mon.a con 0x7f0288005280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940bc6e0 con 0x7f0288005280 2024-09-16T10:57:50.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028800c1d0 con 0x7f0288004870 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f028800c390 con 0x7f0288003a20 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f028800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0288003a20 0x7f0288003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): get_auth_request con 0x7f0288005280 auth_method 0 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:50.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): get_auth_request con 0x7f0288003a20 auth_method 0 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 0x7f0288004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 10 monclient(hunting): get_auth_request con 0x7f0288004870 auth_method 0 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:50.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 1 -- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 msgr2=0x7f0288004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 1 -- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 msgr2=0x7f0288004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 0x7f0288004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a0981700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 0x7f0288004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient(hunting): handle_auth_done global_id 28854 payload 293 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 -- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0288003a20 msgr2=0x7f0288003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0288003a20 0x7f0288003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 -- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 msgr2=0x7f0288004cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0288004870 0x7f0288004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient: _finish_hunting 0 2024-09-16T10:57:50.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 monclient: found mon.a 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 -- 172.21.15.5:0/3485312718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02880035d0 con 0x7f0288005280 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient: _finish_auth 0 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 10 monclient: _check_auth_tickets 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 -- 172.21.15.5:0/3485312718 reap_dead start 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a1983700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f028800ba30 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 crypto rx=0x7f029c04f480 tx=0x7f02880058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a14971bd37ac91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c067310 con 0x7f0288005280 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:50.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: dump: 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c067470 con 0x7f0288005280 2024-09-16T10:57:50.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 4 set_mon_vals no callback set 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 1 -- 172.21.15.5:0/3485312718 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c067620 con 0x7f0288005280 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f0293fff700 10 monclient: dump: 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.954 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 10 monclient: shutdown 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 -- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 msgr2=0x7f028800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f028800ba30 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 crypto rx=0x7f029c04f480 tx=0x7f02880058a0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 -- 172.21.15.5:0/3485312718 shutdown_connections 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 --2- 172.21.15.5:0/3485312718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f028800ba30 unknown :-1 s=CLOSED pgs=6891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 -- 172.21.15.5:0/3485312718 >> 172.21.15.5:0/3485312718 conn(0x7f0294006460 msgr2=0x7f0294006870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 -- 172.21.15.5:0/3485312718 shutdown_connections 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.932+0000 7f02a2184700 1 -- 172.21.15.5:0/3485312718 wait complete. 2024-09-16T10:57:50.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: build_initial_monmap 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: monmap: 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 librados: starting msgr at 2024-09-16T10:57:50.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 librados: starting objecter 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 Processor -- start 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- start start 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 librados: setting wanted keys 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 librados: calling monclient init 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: init 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-a con 0x7f0288005280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0294150f80 0x7f02940a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-c con 0x7f0294150f80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): picked mon.noname-b con 0x7f02941513a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940ab5c0 con 0x7f0288005280 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02940ab810 con 0x7f02941513a0 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02940aa2f0 con 0x7f0294150f80 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient(hunting): authenticate will time out at 1329.513482s 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50978/0 (socket says 172.21.15.5:50978) 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 -- 172.21.15.5:0/408009884 learned_addr learned my addr 172.21.15.5:0/408009884 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): get_auth_request con 0x7f0288005280 auth_method 0 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 10 monclient(hunting): get_auth_request con 0x7f02941513a0 auth_method 0 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a0981700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0294150f80 0x7f02940a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a0981700 10 monclient(hunting): get_auth_request con 0x7f0294150f80 auth_method 0 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a0981700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:50.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a0981700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a0981700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 1 -- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 msgr2=0x7f02940a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 1 -- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 msgr2=0x7f02940a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient(hunting): handle_auth_done global_id 28857 payload 824 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1983700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 -- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0294150f80 msgr2=0x7f02940a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0294150f80 0x7f02940a7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 -- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 msgr2=0x7f02940a80b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient: _finish_hunting 0 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 monclient: found mon.noname-a 2024-09-16T10:57:50.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 -- 172.21.15.5:0/408009884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294023840 con 0x7f0288005280 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a1182700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 secure :-1 s=READY pgs=6892 cs=0 l=1 rev1=1 crypto rx=0x7f0298011630 tx=0x7f02980119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 client.?.objecter ms_handle_connect 0x7f0288005280 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029800c070 con 0x7f0288005280 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: dump: 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.961 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: _finish_auth 0 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 5 monclient: authenticate success, global_id 28857 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0298015870 con 0x7f0288005280 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _renew_subs 2024-09-16T10:57:50.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0294024cb0 con 0x7f0288005280 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0298015c00 con 0x7f0288005280 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02937fe700 4 set_mon_vals no callback set 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 client.28857.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 10 monclient: dump: 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:50.963 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _renew_subs 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02940a9720 con 0x7f0288005280 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 librados: init done 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0298013400 con 0x7f0288005280 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:50.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.936+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02940ab370 con 0x7f0288005280 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 1 --2- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 0x7f0280039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(344..344 src has 57..344) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f0298048020 con 0x7f0288005280 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 10 client.28857.objecter ms_dispatch 0x7f029c000fd0 osd_map(344..344 src has 57..344) v4 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 3 client.28857.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 3 client.28857.objecter handle_osd_map decoding full epoch 344 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 20 client.28857.objecter dump_active .. 0 homeless 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f02a0981700 1 --2- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 0x7f0280039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f02a0981700 10 monclient: get_auth_request con 0x7f0280036f60 auth_method 0 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f0291ffb700 10 client.28857.objecter ms_handle_connect 0x7f0280036f60 2024-09-16T10:57:50.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.940+0000 7f02a0981700 1 --2- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 0x7f0280039420 secure :-1 s=READY pgs=3800 cs=0 l=1 rev1=1 crypto rx=0x7f0294026c60 tx=0x7f028c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:50.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.944+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f029801dc30 con 0x7f0288005280 2024-09-16T10:57:50.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.944+0000 7f0291ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:50.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:50.944+0000 7f0291ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.176+0000 7f02a2184700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.176+0000 7f02a2184700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.176+0000 7f02a2184700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.176+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f02940ab950 con 0x7f0288005280 2024-09-16T10:57:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f0291ffb700 1 -- 172.21.15.5:0/408009884 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f02940ab950 con 0x7f0288005280 2024-09-16T10:57:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f0291ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:51.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f0291ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:51.182 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 10 librados: watch_flush enter 2024-09-16T10:57:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 10 librados: watch_flush exit 2024-09-16T10:57:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 20 client.28857.objecter shutdown clearing up homeless session... 2024-09-16T10:57:51.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 10 client.28857.objecter successfully canceled tick 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 msgr2=0x7f0280039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 0x7f0280039420 secure :-1 s=READY pgs=3800 cs=0 l=1 rev1=1 crypto rx=0x7f0294026c60 tx=0x7f028c009040 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 10 monclient: shutdown 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 msgr2=0x7f0294150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 secure :-1 s=READY pgs=6892 cs=0 l=1 rev1=1 crypto rx=0x7f0298011630 tx=0x7f02980119f0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 shutdown_connections 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02941513a0 0x7f02940a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0294150f80 0x7f02940a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0280036f60 0x7f0280039420 unknown :-1 s=CLOSED pgs=3800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 --2- 172.21.15.5:0/408009884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0288005280 0x7f0294150a40 unknown :-1 s=CLOSED pgs=6892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 >> 172.21.15.5:0/408009884 conn(0x7f0294006460 msgr2=0x7f0294021e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 shutdown_connections 2024-09-16T10:57:51.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 -- 172.21.15.5:0/408009884 wait complete. 2024-09-16T10:57:51.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.180+0000 7f02a2184700 1 librados: shutdown 2024-09-16T10:57:51.195 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:57:51.195 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:57:51.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:57:51.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: get_monmap_and_config 2024-09-16T10:57:51.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: monmap: 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:51.306951+0000 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:51.306951+0000 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:51.308 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 Processor -- start 2024-09-16T10:57:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 -- start start 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: init 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e28071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e2809aa50 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 0x7f7e2813b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-c con 0x7f7e28138f00 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2809caa0 con 0x7f7e2809aa50 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2809e3a0 con 0x7f7e28071870 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2809ce70 con 0x7f7e28138f00 2024-09-16T10:57:51.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2db22700 10 monclient(hunting): authenticate will time out at 1329.883368s 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e27fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7e2809aa50 0x7f7e281377e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43870/0 (socket says 172.21.15.5:43870) 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e27fff700 1 -- 172.21.15.5:0/1407387635 learned_addr learned my addr 172.21.15.5:0/1407387635 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 0x7f7e2813b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2d321700 10 monclient(hunting): get_auth_request con 0x7f7e28138f00 auth_method 0 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2d321700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2d321700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.304+0000 7f7e2d321700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421957122 0 0) 0x7f7e18001490 con 0x7f7e2809aa50 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): _negotiate 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): my global_id is 28860 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e140035d0 con 0x7f7e2809aa50 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 msgr2=0x7f7e2813b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 msgr2=0x7f7e2813b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 0x7f7e2813b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 0x7f7e2813b2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412845907 0 0) 0x7f7e140035d0 con 0x7f7e2809aa50 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 msgr2=0x7f7e2813b2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 --2- 172.21.15.5:0/1407387635 >> v2:172.21.15.143:3300/0 conn(0x7f7e28138f00 0x7f7e2813b2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 >> v1:172.21.15.143:6789/0 conn(0x7f7e28071870 legacy=0x7f7e280a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:57:51.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2809b5f0 con 0x7f7e2809aa50 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f7e18001b80 con 0x7f7e2809aa50 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 5 monclient: authenticate success, global_id 28860 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: dump: 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.317 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 >> v1:172.21.15.5:6789/0 conn(0x7f7e2809aa50 legacy=0x7f7e281377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:57:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e14003a20 0x7f7e14003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): picked mon.b con 0x7f7e14003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e14004870 0x7f7e14004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): picked mon.a con 0x7f7e14004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e1400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): picked mon.c con 0x7f7e14005430 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1400c290 con 0x7f7e14004870 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1400c400 con 0x7f7e14003a20 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1400c5c0 con 0x7f7e14005430 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e1400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e14003a20 0x7f7e14003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request con 0x7f7e14003a20 auth_method 0 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): get_auth_request con 0x7f7e14005430 auth_method 0 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e14004870 0x7f7e14004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): get_auth_request con 0x7f7e14004870 auth_method 0 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient(hunting): handle_auth_done global_id 28860 payload 293 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e14003a20 msgr2=0x7f7e14003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e14003a20 0x7f7e14003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e14004870 msgr2=0x7f7e14004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e14004870 0x7f7e14004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 monclient: found mon.c 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140035d0 con 0x7f7e14005430 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 -- 172.21.15.5:0/1407387635 reap_dead start 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2d321700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e1400ba30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e140058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e20578f82298799 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000c070 con 0x7f7e14005430 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: dump: 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.323 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20013610 con 0x7f7e14005430 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 1 -- 172.21.15.5:0/1407387635 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e200094b0 con 0x7f7e14005430 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 4 set_mon_vals no callback set 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e277fe700 10 monclient: dump: 2024-09-16T10:57:51.324 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 10 monclient: shutdown 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 -- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 msgr2=0x7f7e1400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e1400ba30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e140058a0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 -- 172.21.15.5:0/1407387635 shutdown_connections 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 --2- 172.21.15.5:0/1407387635 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e1400ba30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 -- 172.21.15.5:0/1407387635 >> 172.21.15.5:0/1407387635 conn(0x7f7e280a45d0 msgr2=0x7f7e280a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 -- 172.21.15.5:0/1407387635 shutdown_connections 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.308+0000 7f7e2db22700 1 -- 172.21.15.5:0/1407387635 wait complete. 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: monmap: 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.326 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 librados: starting msgr at 2024-09-16T10:57:51.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 librados: starting objecter 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 Processor -- start 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- start start 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 librados: setting wanted keys 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 librados: calling monclient init 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: init 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e28136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-c con 0x7f7e14005430 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 0x7f7e28137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-a con 0x7f7e28137430 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2811a530 0x7f7e2811a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): picked mon.noname-b con 0x7f7e2811a530 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28137d90 con 0x7f7e28137430 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2811ba00 con 0x7f7e2811a530 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2811db90 con 0x7f7e14005430 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient(hunting): authenticate will time out at 1329.889798s 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 0x7f7e28137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e28136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 0x7f7e28137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50996/0 (socket says 172.21.15.5:50996) 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e28136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48642/0 (socket says 172.21.15.5:48642) 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 -- 172.21.15.5:0/2580119760 learned_addr learned my addr 172.21.15.5:0/2580119760 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): get_auth_request con 0x7f7e28137430 auth_method 0 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request con 0x7f7e14005430 auth_method 0 2024-09-16T10:57:51.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2811a530 0x7f7e2811a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): get_auth_request con 0x7f7e2811a530 auth_method 0 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2d321700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient(hunting): handle_auth_done global_id 28863 payload 824 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 msgr2=0x7f7e28136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e28136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2811a530 msgr2=0x7f7e2811a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2811a530 0x7f7e2811a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 monclient: found mon.noname-a 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2811de70 con 0x7f7e28137430 2024-09-16T10:57:51.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7e14005430 0x7f7e28136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e27fff700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 0x7f7e28137850 secure :-1 s=READY pgs=6895 cs=0 l=1 rev1=1 crypto rx=0x7f7e18001c30 tx=0x7f7e18001c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 client.?.objecter ms_handle_connect 0x7f7e28137430 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18007870 con 0x7f7e28137430 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: dump: 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.339 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 5 monclient: authenticate success, global_id 28863 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e180079f0 con 0x7f7e28137430 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _renew_subs 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e280a3220 con 0x7f7e28137430 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18013630 con 0x7f7e28137430 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e26ffd700 4 set_mon_vals no callback set 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e257fa700 10 monclient: dump: 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.341 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 client.28863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _renew_subs 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2809b0d0 con 0x7f7e28137430 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.312+0000 7f7e2db22700 1 librados: init done 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2db22700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2db22700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 0x7f7e2806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28067380 con 0x7f7e2811aef0 2024-09-16T10:57:51.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 0x7f7e2806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient: get_auth_request con 0x7f7e2811aef0 auth_method 0 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7e18013790 con 0x7f7e28137430 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_done global_id 28866 payload 824 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e2809d580 con 0x7f7e2811aef0 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 0x7f7e2806d330 secure :-1 s=READY pgs=6896 cs=0 l=1 rev1=1 crypto rx=0x7f7e10007fd0 tx=0x7f7e10012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 --2- 172.21.15.5:0/2580119760 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7e08036f60 0x7f7e08039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 client.28863.objecter ms_handle_connect 0x7f7e2811aef0 2024-09-16T10:57:51.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 client.28863.objecter resend_mon_ops 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(344..344 src has 57..344) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f7e18048790 con 0x7f7e28137430 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 client.28863.objecter ms_dispatch 0x7f7e28000fd0 osd_map(344..344 src has 57..344) v4 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 3 client.28863.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 3 client.28863.objecter handle_osd_map decoding full epoch 344 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 20 client.28863.objecter dump_active .. 0 homeless 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e100053f0 con 0x7f7e2811aef0 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 monclient: dump: 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.344 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2d321700 1 --2- 172.21.15.5:0/2580119760 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7e08036f60 0x7f7e08039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2d321700 10 monclient: get_auth_request con 0x7f7e08036f60 auth_method 0 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f7e10005550 con 0x7f7e2811aef0 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:51.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 msgr2=0x7f7e2806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 0x7f7e2806d330 secure :-1 s=READY pgs=6896 cs=0 l=1 rev1=1 crypto rx=0x7f7e10007fd0 tx=0x7f7e10012f60 comp rx=0 tx=0).stop 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 msgr2=0x7f7e2806d330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2811aef0 0x7f7e2806d330 unknown :-1 s=CLOSED pgs=6896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e2d321700 1 --2- 172.21.15.5:0/2580119760 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7e08036f60 0x7f7e08039420 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c750 tx=0x7f7e20020040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.316+0000 7f7e257fa700 10 client.28863.objecter ms_handle_connect 0x7f7e08036f60 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2db22700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2db22700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 0x7f7e2806dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2db22700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2809c250 con 0x7f7e2806d870 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 0x7f7e2806dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient: get_auth_request con 0x7f7e2806d870 auth_method 0 2024-09-16T10:57:51.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 10 monclient(hunting): handle_auth_done global_id 28869 payload 824 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 1 -- 172.21.15.5:0/2580119760 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7e280a0680 con 0x7f7e2806d870 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 10 client.28863.objecter ms_handle_connect 0x7f7e2806d870 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e2cb20700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 0x7f7e2806dc80 secure :-1 s=READY pgs=6897 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009010 tx=0x7f7e10001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 10 client.28863.objecter resend_mon_ops 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10025070 con 0x7f7e2806d870 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.324+0000 7f7e257fa700 10 monclient: dump: 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f7e10007240 con 0x7f7e2806d870 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 msgr2=0x7f7e2806dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 0x7f7e2806dc80 secure :-1 s=READY pgs=6897 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009010 tx=0x7f7e10001120 comp rx=0 tx=0).stop 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 msgr2=0x7f7e2806dc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e257fa700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e2806d870 0x7f7e2806dc80 secure :-1 s=CLOSED pgs=6897 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009010 tx=0x7f7e10001120 comp rx=0 tx=0).stop 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:51.349 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.350 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.351 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:51.352 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.353 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.354 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.355 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.356 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.357 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.358 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.359 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.360 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:51.361 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 10 librados: watch_flush enter 2024-09-16T10:57:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 10 librados: watch_flush exit 2024-09-16T10:57:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 20 client.28863.objecter shutdown clearing up homeless session... 2024-09-16T10:57:51.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 10 client.28863.objecter successfully canceled tick 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7e08036f60 msgr2=0x7f7e08039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 --2- 172.21.15.5:0/2580119760 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7e08036f60 0x7f7e08039420 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400c750 tx=0x7f7e20020040 comp rx=0 tx=0).stop 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 10 monclient: shutdown 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 msgr2=0x7f7e28137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 --2- 172.21.15.5:0/2580119760 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7e28137430 0x7f7e28137850 secure :-1 s=READY pgs=6895 cs=0 l=1 rev1=1 crypto rx=0x7f7e18001c30 tx=0x7f7e18001c60 comp rx=0 tx=0).stop 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2d321700 1 -- 172.21.15.5:0/2580119760 reap_dead start 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2d321700 1 -- 172.21.15.5:0/2580119760 reap_dead start 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 shutdown_connections 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 >> 172.21.15.5:0/2580119760 conn(0x7f7e280a45d0 msgr2=0x7f7e2813b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 shutdown_connections 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 -- 172.21.15.5:0/2580119760 wait complete. 2024-09-16T10:57:51.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.328+0000 7f7e2db22700 1 librados: shutdown 2024-09-16T10:57:51.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:51.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:57:51.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.480+0000 7fe433fff700 10 monclient: get_monmap_and_config 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.480+0000 7fe433fff700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.480+0000 7fe433fff700 10 monclient: monmap: 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:51.484429+0000 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:51.484429+0000 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.485 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:51.486 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 Processor -- start 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- start start 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient: init 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe43409aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe434071870 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe4341376e0 0x7fe434139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe4341376e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43409caa0 con 0x7fe434071870 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43409e3a0 con 0x7fe4341376e0 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43409ae30 con 0x7fe43409aa50 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient(hunting): authenticate will time out at 1330.060919s 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48648/0 (socket says 172.21.15.5:48648) 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe4341376e0 0x7fe434139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 1 -- 172.21.15.5:0/2678637979 learned_addr learned my addr 172.21.15.5:0/2678637979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): get_auth_request con 0x7fe43409aa50 auth_method 0 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe42a7fc700 1 --1- 172.21.15.5:0/2678637979 >> v1:172.21.15.5:6789/0 conn(0x7fe434071870 0x7fe43409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43876/0 (socket says 172.21.15.5:43876) 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): get_auth_request con 0x7fe4341376e0 auth_method 0 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient(hunting): handle_auth_done global_id 28872 payload 274 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 -- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 msgr2=0x7fe43409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 -- 172.21.15.5:0/2678637979 >> v1:172.21.15.5:6789/0 conn(0x7fe434071870 legacy=0x7fe43409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 monclient: found mon.noname-a 2024-09-16T10:57:51.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 -- 172.21.15.5:0/2678637979 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43409b5f0 con 0x7fe4341376e0 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe432ffd700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4337fe700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4341376e0 0x7fe434139ad0 secure :-1 s=READY pgs=6899 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009d90 tx=0x7fe42c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9808ea71828e40cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 1 -- 172.21.15.5:0/2678637979 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c00e070 con 0x7fe4341376e0 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: dump: 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.500 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 1 -- 172.21.15.5:0/2678637979 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c00bb20 con 0x7fe4341376e0 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 5 monclient: authenticate success, global_id 28872 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 1 -- 172.21.15.5:0/2678637979 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c012430 con 0x7fe4341376e0 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 4 set_mon_vals no callback set 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe4327fc700 10 monclient: dump: 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.502 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 10 monclient: shutdown 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4341376e0 msgr2=0x7fe434139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4341376e0 0x7fe434139ad0 secure :-1 s=READY pgs=6899 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009d90 tx=0x7fe42c00b330 comp rx=0 tx=0).stop 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- 172.21.15.5:0/2678637979 shutdown_connections 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe4341376e0 0x7fe434139ad0 unknown :-1 s=CLOSED pgs=6899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 --2- 172.21.15.5:0/2678637979 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe43409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- 172.21.15.5:0/2678637979 >> 172.21.15.5:0/2678637979 conn(0x7fe4340a45d0 msgr2=0x7fe4340a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- 172.21.15.5:0/2678637979 shutdown_connections 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.484+0000 7fe433fff700 1 -- 172.21.15.5:0/2678637979 wait complete. 2024-09-16T10:57:51.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: monmap: 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 librados: starting msgr at 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 librados: starting objecter 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 Processor -- start 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- start start 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 librados: setting wanted keys 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 librados: calling monclient init 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: init 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 0x7fe4341ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe434071870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe434136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe43409aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4341376e0 0x7fe434137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe4341376e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe434120500 con 0x7fe434071870 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4340a2df0 con 0x7fe4341376e0 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43409b0d0 con 0x7fe43409aa50 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient(hunting): authenticate will time out at 1330.065792s 2024-09-16T10:57:51.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 0x7fe4341ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe434136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 0x7fe4341ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51044/0 (socket says 172.21.15.5:51044) 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 -- 172.21.15.5:0/3409502106 learned_addr learned my addr 172.21.15.5:0/3409502106 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): get_auth_request con 0x7fe434071870 auth_method 0 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): get_auth_request con 0x7fe43409aa50 auth_method 0 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe4337fe700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4341376e0 0x7fe434137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient(hunting): handle_auth_done global_id 28878 payload 824 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 msgr2=0x7fe434136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43409aa50 0x7fe434136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4341376e0 msgr2=0x7fe434137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4341376e0 0x7fe434137030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 monclient: found mon.noname-a 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe424009570 con 0x7fe434071870 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe4337fe700 10 monclient: get_auth_request con 0x7fe4341376e0 auth_method 0 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe4337fe700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4341376e0 0x7fe434137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe432ffd700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 0x7fe4341ca440 secure :-1 s=READY pgs=6900 cs=0 l=1 rev1=1 crypto rx=0x7fe42400ebe0 tx=0x7fe42400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 client.?.objecter ms_handle_connect 0x7fe434071870 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42400c850 con 0x7fe434071870 2024-09-16T10:57:51.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: dump: 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 5 monclient: authenticate success, global_id 28878 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42400c9b0 con 0x7fe434071870 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _renew_subs 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c009a70 con 0x7fe434071870 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42400cd20 con 0x7fe434071870 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe431ffb700 4 set_mon_vals no callback set 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe42bfff700 10 monclient: dump: 2024-09-16T10:57:51.511 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 client.28878.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _renew_subs 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:51.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43411feb0 con 0x7fe434071870 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.488+0000 7fe433fff700 1 librados: init done 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe424013470 con 0x7fe434071870 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe433fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe433fff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 0x7fe43402b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe434067380 con 0x7fe43411cc90 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 --2- 172.21.15.5:0/3409502106 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe414036f60 0x7fe414039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(344..344 src has 57..344) v4 ==== 5317+0+0 (secure 0 0 0) 0x7fe424047ee0 con 0x7fe434071870 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 client.28878.objecter ms_dispatch 0x7fe434000fd0 osd_map(344..344 src has 57..344) v4 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 3 client.28878.objecter handle_osd_map got epochs [344,344] > 0 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 3 client.28878.objecter handle_osd_map decoding full epoch 344 2024-09-16T10:57:51.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 0x7fe43402b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient: get_auth_request con 0x7fe43411cc90 auth_method 0 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 20 client.28878.objecter dump_active .. 0 homeless 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe4337fe700 1 --2- 172.21.15.5:0/3409502106 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe414036f60 0x7fe414039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe4337fe700 10 monclient: get_auth_request con 0x7fe414036f60 auth_method 0 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_done global_id 54260 payload 824 2024-09-16T10:57:51.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe43409d580 con 0x7fe43411cc90 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 client.28878.objecter ms_handle_connect 0x7fe43411cc90 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42a7fc700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 0x7fe43402b6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe43411df40 tx=0x7fe41c001870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 client.28878.objecter resend_mon_ops 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c017070 con 0x7fe43411cc90 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 monclient: dump: 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.515 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 client.28878.objecter ms_handle_connect 0x7fe414036f60 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe4337fe700 1 --2- 172.21.15.5:0/3409502106 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe414036f60 0x7fe414039420 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7fe43411ed20 tx=0x7fe42c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe41c012560 con 0x7fe43411cc90 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:51.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 msgr2=0x7fe43402b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 0x7fe43402b6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe43411df40 tx=0x7fe41c001870 comp rx=0 tx=0).stop 2024-09-16T10:57:51.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 msgr2=0x7fe43402b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.492+0000 7fe42bfff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411cc90 0x7fe43402b6a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 0x7fe43402bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43409c210 con 0x7fe43411d070 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 0x7fe43402bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient: get_auth_request con 0x7fe43411d070 auth_method 0 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 10 monclient(hunting): handle_auth_done global_id 54263 payload 824 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 1 -- 172.21.15.5:0/3409502106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe4340a0680 con 0x7fe43411d070 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 client.28878.objecter ms_handle_connect 0x7fe43411d070 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 client.28878.objecter resend_mon_ops 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42a7fc700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 0x7fe43402bbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe43411df40 tx=0x7fe41c001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41c023070 con 0x7fe43411d070 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 monclient: dump: 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.520 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7fe41c012e60 con 0x7fe43411d070 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 msgr2=0x7fe43402bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 0x7fe43402bbe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe43411df40 tx=0x7fe41c001080 comp rx=0 tx=0).stop 2024-09-16T10:57:51.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 msgr2=0x7fe43402bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe42bfff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe43411d070 0x7fe43402bbe0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe43411df40 tx=0x7fe41c001080 comp rx=0 tx=0).stop 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:51.522 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.523 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.524 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:51.525 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.526 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.527 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.528 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.529 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:51.530 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.532 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.533 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:51.534 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 librados: watch_flush enter 2024-09-16T10:57:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 librados: watch_flush exit 2024-09-16T10:57:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 20 client.28878.objecter shutdown clearing up homeless session... 2024-09-16T10:57:51.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 client.28878.objecter successfully canceled tick 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe414036f60 msgr2=0x7fe414039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 --2- 172.21.15.5:0/3409502106 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe414036f60 0x7fe414039420 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7fe43411ed20 tx=0x7fe42c011000 comp rx=0 tx=0).stop 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 10 monclient: shutdown 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 msgr2=0x7fe4341ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 --2- 172.21.15.5:0/3409502106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe434071870 0x7fe4341ca440 secure :-1 s=READY pgs=6900 cs=0 l=1 rev1=1 crypto rx=0x7fe42400ebe0 tx=0x7fe42400eef0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe4337fe700 1 -- 172.21.15.5:0/3409502106 reap_dead start 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe4337fe700 1 -- 172.21.15.5:0/3409502106 reap_dead start 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 shutdown_connections 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 >> 172.21.15.5:0/3409502106 conn(0x7fe4340a45d0 msgr2=0x7fe434099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 shutdown_connections 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 -- 172.21.15.5:0/3409502106 wait complete. 2024-09-16T10:57:51.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.504+0000 7fe433fff700 1 librados: shutdown 2024-09-16T10:57:51.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:51.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:57:51.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: get_monmap_and_config 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: monmap: 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:51.661261+0000 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:51.661261+0000 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:51.663 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 Processor -- start 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 -- start start 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: init 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T10:57:51.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff83c09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff83c09d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff83c09dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c070e40 con 0x7ff83c09dfd0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0a01d0 con 0x7ff83c09d6a0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c09f740 con 0x7ff83c09e8e0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient(hunting): authenticate will time out at 1330.237927s 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48680/0 (socket says 172.21.15.5:48680) 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83b7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff83c09dfd0 0x7ff83c1336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43892/0 (socket says 172.21.15.5:43892) 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 -- 172.21.15.5:0/3426925628 learned_addr learned my addr 172.21.15.5:0/3426925628 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): get_auth_request con 0x7ff83c09e8e0 auth_method 0 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 1 --2- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 10 monclient(hunting): get_auth_request con 0x7ff83c09d6a0 auth_method 0 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 1 -- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 msgr2=0x7ff83c09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487132544 0 0) 0x7ff83c070e40 con 0x7ff83c09dfd0 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 1 -- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 msgr2=0x7ff83c09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient(hunting): _negotiate 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 1 --2- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient(hunting): my global_id is 28881 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8280035d0 con 0x7ff83c09dfd0 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff8327fc700 1 --2- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient(hunting): handle_auth_done global_id 54266 payload 274 2024-09-16T10:57:51.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749802536 0 0) 0x7ff8280035d0 con 0x7ff83c09dfd0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 -- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 msgr2=0x7ff83c09f200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 --2- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 -- 172.21.15.5:0/3426925628 >> v1:172.21.15.5:6789/0 conn(0x7ff83c09dfd0 legacy=0x7ff83c1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 monclient: found mon.noname-b 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 -- 172.21.15.5:0/3426925628 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c09fef0 con 0x7ff83c09e8e0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83affd700 1 --2- 172.21.15.5:0/3426925628 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009910 tx=0x7ff82c009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55ff42b97aef94ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340019f0 con 0x7ff83c09e8e0 2024-09-16T10:57:51.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: dump: 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 5 monclient: authenticate success, global_id 54266 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c004030 con 0x7ff83c09e8e0 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 1 -- 172.21.15.5:0/3426925628 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c0043c0 con 0x7ff83c09e8e0 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83bfff700 4 set_mon_vals no callback set 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.660+0000 7ff83a7fc700 10 monclient: dump: 2024-09-16T10:57:51.673 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: shutdown 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- 172.21.15.5:0/3426925628 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 msgr2=0x7ff83c09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- 172.21.15.5:0/3426925628 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009910 tx=0x7ff82c009c20 comp rx=0 tx=0).stop 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- 172.21.15.5:0/3426925628 shutdown_connections 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- 172.21.15.5:0/3426925628 >> v2:172.21.15.143:3300/0 conn(0x7ff83c09d6a0 0x7ff83c09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- 172.21.15.5:0/3426925628 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09e8e0 0x7ff83c09ecc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- 172.21.15.5:0/3426925628 >> 172.21.15.5:0/3426925628 conn(0x7ff83c0a45d0 msgr2=0x7ff83c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- 172.21.15.5:0/3426925628 shutdown_connections 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- 172.21.15.5:0/3426925628 wait complete. 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: build_initial_monmap 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: monmap: 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:51.675 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 librados: starting msgr at 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 librados: starting objecter 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 Processor -- start 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- start start 2024-09-16T10:57:51.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 librados: setting wanted keys 2024-09-16T10:57:51.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 librados: calling monclient init 2024-09-16T10:57:51.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: init 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-c con 0x7ff83c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff83c132aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-b con 0x7ff83c09dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff83c11c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): picked mon.noname-a con 0x7ff83c09e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c1c90b0 con 0x7ff83c09e8e0 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c070370 con 0x7ff83c09dfd0 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c09b0d0 con 0x7ff83c09d6a0 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83bfff700 10 monclient(hunting): authenticate will time out at 1330.243033s 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:48694/0 (socket says 172.21.15.5:48694) 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 learned_addr learned my addr 172.21.15.5:0/4186039950 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): get_auth_request con 0x7ff83c09d6a0 auth_method 0 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff83c11c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff83c132aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): get_auth_request con 0x7ff83c09e8e0 auth_method 0 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 10 monclient(hunting): get_auth_request con 0x7ff83c09dfd0 auth_method 0 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient(hunting): handle_auth_done global_id 54269 payload 824 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 msgr2=0x7ff83c132aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff83c132aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 msgr2=0x7ff83c11c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff83c11c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 monclient: found mon.noname-c 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009570 con 0x7ff83c09d6a0 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff8327fc700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff83c132aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 msgr2=0x7ff83c1c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 msgr2=0x7ff83c1c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.664+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff82c009f70 tx=0x7ff82c004770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:57:51.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.668+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient: get_auth_request con 0x7ff83c09d6a0 auth_method 0 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 0 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 msgr2=0x7ff83c1c8140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff83c09d6a0 0x7ff83c1c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:57:51.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff81c0055b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff83c09e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff81c008040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff83c09dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 0x7ff81c00af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff81c008aa0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c00ba30 con 0x7ff83c09e8e0 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c00bb70 con 0x7ff83c09dfd0 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c00bd30 con 0x7ff81c008aa0 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:51.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 0x7ff81c00af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): get_auth_request con 0x7ff81c008aa0 auth_method 0 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff81c0055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff81c008040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): get_auth_request con 0x7ff83c09e8e0 auth_method 0 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): get_auth_request con 0x7ff83c09dfd0 auth_method 0 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff8327fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient(hunting): handle_auth_done global_id 54269 payload 293 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 msgr2=0x7ff81c008040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c09dfd0 0x7ff81c008040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 msgr2=0x7ff81c0055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff83c09e8e0 0x7ff81c0055b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 monclient: found mon.noname-c 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c00bfb0 con 0x7ff81c008aa0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83b7fe700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 0x7ff81c00af30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8340012e0 tx=0x7ff82c008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 client.?.objecter ms_handle_connect 0x7ff81c008aa0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff834001860 con 0x7ff81c008aa0 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: dump: 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: _finish_auth 0 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: _check_auth_tickets 2024-09-16T10:57:51.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 5 monclient: authenticate success, global_id 54269 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 10 monclient: _renew_subs 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c11f120 con 0x7ff81c008aa0 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8340133f0 con 0x7ff81c008aa0 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340137a0 con 0x7ff81c008aa0 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff833fff700 10 monclient: dump: 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff839ffb700 4 set_mon_vals no callback set 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 10 client.54269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 10 monclient: _renew_subs 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c120460 con 0x7ff81c008aa0 2024-09-16T10:57:51.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.872+0000 7ff83bfff700 1 librados: init done 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83bfff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 0x7ff83c132aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c067380 con 0x7ff83c1c8140 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff834013900 con 0x7ff81c008aa0 2024-09-16T10:57:51.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 0x7ff83c132aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient: get_auth_request con 0x7ff83c1c8140 auth_method 0 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff83c09d6a0 0x7ff81c041690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(345..345 src has 57..345) v4 ==== 4932+0+0 (secure 0 0 0) 0x7ff834047380 con 0x7ff81c008aa0 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 client.54269.objecter ms_dispatch 0x7ff83c000fd0 osd_map(345..345 src has 57..345) v4 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 3 client.54269.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 3 client.54269.objecter handle_osd_map decoding full epoch 345 2024-09-16T10:57:51.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 10 monclient(hunting): handle_auth_done global_id 64108 payload 824 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff83c06cc90 con 0x7ff83c1c8140 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff8327fc700 1 --2- 172.21.15.5:0/4186039950 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff83c09d6a0 0x7ff81c041690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 20 client.54269.objecter dump_active .. 0 homeless 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff8327fc700 10 monclient: get_auth_request con 0x7ff83c09d6a0 auth_method 0 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 client.54269.objecter ms_handle_connect 0x7ff83c1c8140 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 client.54269.objecter resend_mon_ops 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 0x7ff83c132aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff83c09b260 tx=0x7ff82c0092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c028890 con 0x7ff83c1c8140 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 monclient: dump: 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.889 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff833fff700 10 client.54269.objecter ms_handle_connect 0x7ff83c09d6a0 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.876+0000 7ff8327fc700 1 --2- 172.21.15.5:0/4186039950 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff83c09d6a0 0x7ff81c041690 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0716b0 tx=0x7ff824008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff82c0289f0 con 0x7ff83c1c8140 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:51.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:51.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 msgr2=0x7ff83c132aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 0x7ff83c132aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff83c09b260 tx=0x7ff82c0092e0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 msgr2=0x7ff83c132aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.880+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c1c8140 0x7ff83c132aa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:51.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 0x7ff83c11c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:51.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:51.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c11fe00 con 0x7ff83c132fe0 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 0x7ff83c11c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient: get_auth_request con 0x7ff83c132fe0 auth_method 0 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 10 monclient(hunting): handle_auth_done global_id 64111 payload 824 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 1 -- 172.21.15.5:0/4186039950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff83c133dd0 con 0x7ff83c132fe0 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83affd700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 0x7ff83c11c4f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0047b0 tx=0x7ff82c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:51.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 client.54269.objecter ms_handle_connect 0x7ff83c132fe0 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 client.54269.objecter resend_mon_ops 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c031800 con 0x7ff83c132fe0 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 monclient: dump: 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:51.898 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7ff82c01de80 con 0x7ff83c132fe0 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 msgr2=0x7ff83c11c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 0x7ff83c11c4f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0047b0 tx=0x7ff82c019000 comp rx=0 tx=0).stop 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 msgr2=0x7ff83c11c4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff833fff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff83c132fe0 0x7ff83c11c4f0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff82c0047b0 tx=0x7ff82c019000 comp rx=0 tx=0).stop 2024-09-16T10:57:51.899 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.900 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:51.901 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.902 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:51.903 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:51.904 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:51.905 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.906 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:51.907 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:51.908 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.909 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:51.910 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:51.911 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:51.912 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 librados: watch_flush enter 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 librados: watch_flush exit 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 20 client.54269.objecter shutdown clearing up homeless session... 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 client.54269.objecter successfully canceled tick 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff83c09d6a0 msgr2=0x7ff81c041690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 --2- 172.21.15.5:0/4186039950 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff83c09d6a0 0x7ff81c041690 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0716b0 tx=0x7ff824008040 comp rx=0 tx=0).stop 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 10 monclient: shutdown 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83b7fe700 1 -- 172.21.15.5:0/4186039950 reap_dead start 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 msgr2=0x7ff81c00af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:51.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 0x7ff81c00af30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff8340012e0 tx=0x7ff82c008030 comp rx=0 tx=0).stop 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 shutdown_connections 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 --2- 172.21.15.5:0/4186039950 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff81c008aa0 0x7ff81c00af30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 >> 172.21.15.5:0/4186039950 conn(0x7ff83c0a45d0 msgr2=0x7ff83c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 shutdown_connections 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 -- 172.21.15.5:0/4186039950 wait complete. 2024-09-16T10:57:51.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:51.888+0000 7ff83bfff700 1 librados: shutdown 2024-09-16T10:57:51.916 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:51.916 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:57:51.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:57:52.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: get_monmap_and_config 2024-09-16T10:57:52.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: build_initial_monmap 2024-09-16T10:57:52.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: monmap: 2024-09-16T10:57:52.035 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:52.035 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:52.035 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:52.033827+0000 2024-09-16T10:57:52.035 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:52.033827+0000 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:52.036 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 Processor -- start 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 -- start start 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: init 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f94a009e8e0 0x7f94a009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-a con 0x7f94a009e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f94a009d6a0 0x7f94a009f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-c con 0x7f94a009d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94a009dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a0070e40 con 0x7f94a009dfd0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a00a01d0 con 0x7f94a009e8e0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a009f740 con 0x7f94a009d6a0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient(hunting): authenticate will time out at 1330.610205s 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f94a009e8e0 0x7f94a009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f94a009e8e0 0x7f94a009ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51074/0 (socket says 172.21.15.5:51074) 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 -- 172.21.15.5:0/2325725669 learned_addr learned my addr 172.21.15.5:0/2325725669 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:52.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): get_auth_request con 0x7f94a009e8e0 auth_method 0 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 1 --2- 172.21.15.5:0/2325725669 >> v2:172.21.15.143:3300/0 conn(0x7f94a009d6a0 0x7f94a009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): get_auth_request con 0x7f94a009d6a0 auth_method 0 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:52.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient(hunting): handle_auth_done global_id 28887 payload 274 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 -- 172.21.15.5:0/2325725669 >> v2:172.21.15.143:3300/0 conn(0x7f94a009d6a0 msgr2=0x7f94a009f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 --2- 172.21.15.5:0/2325725669 >> v2:172.21.15.143:3300/0 conn(0x7f94a009d6a0 0x7f94a009f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 -- 172.21.15.5:0/2325725669 >> v1:172.21.15.143:6789/0 conn(0x7f94a009dfd0 legacy=0x7f94a01336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 monclient: found mon.noname-a 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 -- 172.21.15.5:0/2325725669 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a009fef0 con 0x7f94a009e8e0 2024-09-16T10:57:52.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949ffff700 1 --2- 172.21.15.5:0/2325725669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009e8e0 0x7f94a009ecc0 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 crypto rx=0x7f9490009910 tx=0x7f9490009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=507ac8801818ce26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 1 -- 172.21.15.5:0/2325725669 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490011830 con 0x7f94a009e8e0 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: dump: 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:52.048 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: _finish_auth 0 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: _check_auth_tickets 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 5 monclient: authenticate success, global_id 28887 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 1 -- 172.21.15.5:0/2325725669 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9490011990 con 0x7f94a009e8e0 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f94a5346700 4 set_mon_vals no callback set 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 1 -- 172.21.15.5:0/2325725669 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490011d40 con 0x7f94a009e8e0 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:52.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.032+0000 7f949effd700 10 monclient: dump: 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: shutdown 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:52.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2325725669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009e8e0 msgr2=0x7f94a009ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- 172.21.15.5:0/2325725669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009e8e0 0x7f94a009ecc0 secure :-1 s=READY pgs=6904 cs=0 l=1 rev1=1 crypto rx=0x7f9490009910 tx=0x7f9490009c20 comp rx=0 tx=0).stop 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2325725669 shutdown_connections 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- 172.21.15.5:0/2325725669 >> v2:172.21.15.143:3300/0 conn(0x7f94a009d6a0 0x7f94a009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- 172.21.15.5:0/2325725669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009e8e0 0x7f94a009ecc0 unknown :-1 s=CLOSED pgs=6904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2325725669 >> 172.21.15.5:0/2325725669 conn(0x7f94a00a45d0 msgr2=0x7f94a00a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2325725669 shutdown_connections 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2325725669 wait complete. 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: build_initial_monmap 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: monmap: 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:52.051 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 librados: starting msgr at 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 librados: starting objecter 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 Processor -- start 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- start start 2024-09-16T10:57:52.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 librados: setting wanted keys 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 librados: calling monclient init 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: init 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-a con 0x7f94a009d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94a009dfd0 0x7f94a0132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-b con 0x7f94a009dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 0x7f94a01ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): picked mon.noname-c con 0x7f94a009e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01c7a60 con 0x7f94a009d6a0 2024-09-16T10:57:52.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a01227c0 con 0x7f94a009dfd0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0070290 con 0x7f94a009e8e0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient(hunting): authenticate will time out at 1330.614423s 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51078/0 (socket says 172.21.15.5:51078) 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 0x7f94a01ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 -- 172.21.15.5:0/2041813621 learned_addr learned my addr 172.21.15.5:0/2041813621 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): get_auth_request con 0x7f94a009d6a0 auth_method 0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): get_auth_request con 0x7f94a009e8e0 auth_method 0 2024-09-16T10:57:52.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94a009dfd0 0x7f94a0132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): get_auth_request con 0x7f94a009dfd0 auth_method 0 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient(hunting): handle_auth_done global_id 28890 payload 824 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 -- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 msgr2=0x7f94a01ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 0x7f94a01ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 -- 172.21.15.5:0/2041813621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94a009dfd0 msgr2=0x7f94a0132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94a009dfd0 0x7f94a0132de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 monclient: found mon.noname-a 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 -- 172.21.15.5:0/2041813621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9490009570 con 0x7f94a009d6a0 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a4b45700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 0x7f94a01ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949ffff700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 secure :-1 s=READY pgs=6905 cs=0 l=1 rev1=1 crypto rx=0x7f94a00a0940 tx=0x7f9490004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 client.?.objecter ms_handle_connect 0x7f94a009d6a0 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490004510 con 0x7f94a009d6a0 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: dump: 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:52.057 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: _finish_auth 0 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: _check_auth_tickets 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 5 monclient: authenticate success, global_id 28890 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _renew_subs 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94900036c0 con 0x7f94a009d6a0 2024-09-16T10:57:52.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a009b0d0 con 0x7f94a009d6a0 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9490003a30 con 0x7f94a009d6a0 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 client.28890.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949cff9700 10 monclient: dump: 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:52.059 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f949e7fc700 4 set_mon_vals no callback set 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _renew_subs 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0121030 con 0x7f94a009d6a0 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.036+0000 7f94a5346700 1 librados: init done 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f94a5346700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f94a5346700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f94a5346700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a0134160 con 0x7f94a009d6a0 2024-09-16T10:57:52.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9490021070 con 0x7f94a009d6a0 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 1 --2- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 0x7f948c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(345..345 src has 57..345) v4 ==== 4932+0+0 (secure 0 0 0) 0x7f9490049080 con 0x7f94a009d6a0 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 10 client.28890.objecter ms_dispatch 0x7f94a0000fd0 osd_map(345..345 src has 57..345) v4 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 3 client.28890.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 3 client.28890.objecter handle_osd_map decoding full epoch 345 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 20 client.28890.objecter dump_active .. 0 homeless 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949f7fe700 1 --2- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 0x7f948c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949f7fe700 10 monclient: get_auth_request con 0x7f948c036f60 auth_method 0 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949cff9700 10 client.28890.objecter ms_handle_connect 0x7f948c036f60 2024-09-16T10:57:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.040+0000 7f949f7fe700 1 --2- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 0x7f948c039420 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f94a00a3230 tx=0x7f9494009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:52.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.048+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f949002d760 con 0x7f94a009d6a0 2024-09-16T10:57:52.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.048+0000 7f949cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:52.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.048+0000 7f949cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:52.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.284+0000 7f94a5346700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:57:52.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.284+0000 7f94a5346700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:57:52.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.284+0000 7f94a5346700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:52.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.284+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f94a006cea0 con 0x7f94a009d6a0 2024-09-16T10:57:52.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f949cff9700 1 -- 172.21.15.5:0/2041813621 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9490015080 con 0x7f94a009d6a0 2024-09-16T10:57:52.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f949cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:57:52.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f949cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=2390520519,config=692619927,health=4053367063,kv=3977292642,logm=2383926468}) 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=9,logm=67} crc {auth=1937863143,config=692619927,health=1464655568,kv=1359114196,logm=3402812244}) 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=2390520519,config=692619927,health=4053367063,kv=3977292642,logm=2383926468}) 2024-09-16T10:57:52.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.288+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=10,logm=64} crc {auth=3613760816,config=692619927,health=1791106926,kv=564032378,logm=2932274908}) 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 10 librados: watch_flush enter 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 10 librados: watch_flush exit 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 20 client.28890.objecter shutdown clearing up homeless session... 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 10 client.28890.objecter successfully canceled tick 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 msgr2=0x7f948c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 0x7f948c039420 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f94a00a3230 tx=0x7f9494009240 comp rx=0 tx=0).stop 2024-09-16T10:57:52.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 10 monclient: shutdown 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 msgr2=0x7f94a01c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 secure :-1 s=READY pgs=6905 cs=0 l=1 rev1=1 crypto rx=0x7f94a00a0940 tx=0x7f9490004770 comp rx=0 tx=0).stop 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 shutdown_connections 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f94a009e8e0 0x7f94a01ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94a009dfd0 0x7f94a0132de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> v2:172.21.15.143:6812/1098205034 conn(0x7f948c036f60 0x7f948c039420 unknown :-1 s=CLOSED pgs=3805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 --2- 172.21.15.5:0/2041813621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f94a009d6a0 0x7f94a01c7000 unknown :-1 s=CLOSED pgs=6905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 >> 172.21.15.5:0/2041813621 conn(0x7f94a00a45d0 msgr2=0x7f94a009f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 shutdown_connections 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 -- 172.21.15.5:0/2041813621 wait complete. 2024-09-16T10:57:52.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f94a5346700 1 librados: shutdown 2024-09-16T10:57:52.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782196830}) 2024-09-16T10:57:52.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2153050674}) 2024-09-16T10:57:52.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782196830}) 2024-09-16T10:57:52.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.292+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812506920}) 2024-09-16T10:57:52.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640587462}) 2024-09-16T10:57:52.299 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748827248}) 2024-09-16T10:57:52.299 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.299 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640587462}) 2024-09-16T10:57:52.299 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251176592}) 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830812487}) 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1014016951}) 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830812487}) 2024-09-16T10:57:52.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.296+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037903639}) 2024-09-16T10:57:52.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390356474}) 2024-09-16T10:57:52.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179843068}) 2024-09-16T10:57:52.303 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.303 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390356474}) 2024-09-16T10:57:52.303 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791234355}) 2024-09-16T10:57:52.303 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.303 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070961781}) 2024-09-16T10:57:52.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555957363}) 2024-09-16T10:57:52.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070961781}) 2024-09-16T10:57:52.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196421112}) 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804329062}) 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558847922}) 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804329062}) 2024-09-16T10:57:52.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.300+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868853372}) 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436921233}) 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549587550}) 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436921233}) 2024-09-16T10:57:52.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=17453110}) 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928428965}) 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544399564}) 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928428965}) 2024-09-16T10:57:52.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849555786}) 2024-09-16T10:57:52.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:57:52.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410469765}) 2024-09-16T10:57:52.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3364522767}) 2024-09-16T10:57:52.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410469765}) 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.304+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597130318}) 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704525198}) 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3440540595}) 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704525198}) 2024-09-16T10:57:52.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437546765}) 2024-09-16T10:57:52.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674815784}) 2024-09-16T10:57:52.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3821117587}) 2024-09-16T10:57:52.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674815784}) 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98421268}) 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146959314}) 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420694302}) 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146959314}) 2024-09-16T10:57:52.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3118561265}) 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019322276}) 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531090832}) 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019322276}) 2024-09-16T10:57:52.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.308+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209350970}) 2024-09-16T10:57:52.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893611290}) 2024-09-16T10:57:52.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549347351}) 2024-09-16T10:57:52.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893611290}) 2024-09-16T10:57:52.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.312+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=138748688}) 2024-09-16T10:57:52.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292421109}) 2024-09-16T10:57:52.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991015055}) 2024-09-16T10:57:52.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292421109}) 2024-09-16T10:57:52.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219494204}) 2024-09-16T10:57:52.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581455561}) 2024-09-16T10:57:52.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049296335}) 2024-09-16T10:57:52.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581455561}) 2024-09-16T10:57:52.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043191965}) 2024-09-16T10:57:52.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615343025}) 2024-09-16T10:57:52.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3920360536}) 2024-09-16T10:57:52.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615343025}) 2024-09-16T10:57:52.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.316+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3961778060}) 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969444029}) 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2274339481}) 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969444029}) 2024-09-16T10:57:52.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633181951}) 2024-09-16T10:57:52.323 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579274779}) 2024-09-16T10:57:52.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138431207}) 2024-09-16T10:57:52.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579274779}) 2024-09-16T10:57:52.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1225580547}) 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952473377}) 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680264470}) 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952473377}) 2024-09-16T10:57:52.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.320+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352147289}) 2024-09-16T10:57:52.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170665528}) 2024-09-16T10:57:52.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3390131696}) 2024-09-16T10:57:52.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170665528}) 2024-09-16T10:57:52.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815343651}) 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417478759}) 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594961214}) 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3417478759}) 2024-09-16T10:57:52.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315260508}) 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823167741}) 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880067910}) 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.324+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=823167741}) 2024-09-16T10:57:52.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164321008}) 2024-09-16T10:57:52.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018818324}) 2024-09-16T10:57:52.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416358438}) 2024-09-16T10:57:52.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018818324}) 2024-09-16T10:57:52.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932147319}) 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950726923}) 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139468957}) 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950726923}) 2024-09-16T10:57:52.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961180281}) 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433037138}) 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818839765}) 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433037138}) 2024-09-16T10:57:52.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.328+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286438930}) 2024-09-16T10:57:52.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953173474}) 2024-09-16T10:57:52.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136091629}) 2024-09-16T10:57:52.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953173474}) 2024-09-16T10:57:52.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923787701}) 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837344154}) 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932500252}) 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837344154}) 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177377535}) 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318370179}) 2024-09-16T10:57:52.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788253740}) 2024-09-16T10:57:52.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318370179}) 2024-09-16T10:57:52.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520326727}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942562317}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3882873156}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3942562317}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.332+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2745032488}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267812562}) 2024-09-16T10:57:52.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018798506}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267812562}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1145616443}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929200607}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=213718191}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929200607}) 2024-09-16T10:57:52.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4008610007}) 2024-09-16T10:57:52.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308540262}) 2024-09-16T10:57:52.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302999515}) 2024-09-16T10:57:52.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308540262}) 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188870083}) 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.336+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=39,osdmap=2} crc {logm=163860054,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=227735332,mon_config_key=3464617691,monmap=2357222013,osd_metadata=2023718583,osd_pg_creating=3693105497,osd_snap=3144779553,osdmap=1827456125}) 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38} crc {logm=1523862037,mdsmap=1531151339,mgr=2803924567,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=3398002416,mon_config_key=2138394017,monmap=2357222013,osd_metadata=1646985115,osd_pg_creating=3693105497,osd_snap=1036581792}) 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=39,osdmap=2} crc {logm=163860054,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=227735332,mon_config_key=3464617691,monmap=2357222013,osd_metadata=2023718583,osd_pg_creating=3693105497,osd_snap=3144779553,osdmap=1827456125}) 2024-09-16T10:57:52.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=3095028106,mdsmap=1531151339,mgr=618435964,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=3398002416,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3693105497,osd_snap=2843808074}) 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4068932274}) 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=80115713}) 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4068932274}) 2024-09-16T10:57:52.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1042939196}) 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4110904200}) 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2459093948}) 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4110904200}) 2024-09-16T10:57:52.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.340+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4190387252}) 2024-09-16T10:57:52.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2499632005}) 2024-09-16T10:57:52.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4180311692}) 2024-09-16T10:57:52.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2499632005}) 2024-09-16T10:57:52.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3866451650}) 2024-09-16T10:57:52.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2872027349}) 2024-09-16T10:57:52.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1414424293}) 2024-09-16T10:57:52.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2872027349}) 2024-09-16T10:57:52.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.344+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3089493118}) 2024-09-16T10:57:52.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=3335847628}) 2024-09-16T10:57:52.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=31} crc {osdmap=149360169}) 2024-09-16T10:57:52.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:57:52.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=3335847628}) 2024-09-16T10:57:52.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T10:57:52.352+0000 7f7f7cbab700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=31} crc {osdmap=149360169}) 2024-09-16T10:57:53.310 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:57:53.310 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:53.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: get_monmap_and_config 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: build_initial_monmap 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: monmap: 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:53.427272+0000 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:53.427272+0000 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:53.428 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:53.429 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 Processor -- start 2024-09-16T10:57:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 -- start start 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: init 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f4c09aa50 0x7f1f4c09e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c09aa50 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f4c071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f4c1376e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c09caa0 con 0x7f1f4c1376e0 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c09ae30 con 0x7f1f4c071870 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c09e3e0 con 0x7f1f4c09aa50 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f50937700 10 monclient(hunting): authenticate will time out at 1332.003728s 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.424+0000 7f1f4b59e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1f4c1376e0 0x7f1f4c139ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46904/0 (socket says 172.21.15.5:46904) 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4b59e700 1 -- 172.21.15.5:0/2639934782 learned_addr learned my addr 172.21.15.5:0/2639934782 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 --2- 172.21.15.5:0/2639934782 >> v2:172.21.15.143:3300/0 conn(0x7f1f4c09aa50 0x7f1f4c09e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): get_auth_request con 0x7f1f4c09aa50 auth_method 0 2024-09-16T10:57:53.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 1 -- 172.21.15.5:0/2639934782 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439619728 0 0) 0x7f1f44001110 con 0x7f1f4c1376e0 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient(hunting): _negotiate 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient(hunting): my global_id is 28893 2024-09-16T10:57:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 1 -- 172.21.15.5:0/2639934782 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f380035d0 con 0x7f1f4c1376e0 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_done global_id 64123 payload 274 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2639934782 >> v1:172.21.15.143:6789/0 conn(0x7f1f4c071870 legacy=0x7f1f4c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2639934782 >> v1:172.21.15.5:6789/0 conn(0x7f1f4c1376e0 legacy=0x7f1f4c139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient: _finish_hunting 0 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 monclient: found mon.noname-c 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2639934782 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c09b5f0 con 0x7f1f4c09aa50 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f4ad9d700 1 --2- 172.21.15.5:0/2639934782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c09e8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c09e520 tx=0x7f1f3c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5edeca316e13c0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 1 -- 172.21.15.5:0/2639934782 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c00f070 con 0x7f1f4c09aa50 2024-09-16T10:57:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: dump: 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: _finish_auth 0 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 5 monclient: authenticate success, global_id 64123 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 1 -- 172.21.15.5:0/2639934782 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c00a2e0 con 0x7f1f4c09aa50 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 1 -- 172.21.15.5:0/2639934782 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c003c40 con 0x7f1f4c09aa50 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 4 set_mon_vals no callback set 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f49d9b700 10 monclient: dump: 2024-09-16T10:57:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 10 monclient: shutdown 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- 172.21.15.5:0/2639934782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 msgr2=0x7f1f4c09e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 --2- 172.21.15.5:0/2639934782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c09e8e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c09e520 tx=0x7f1f3c00c330 comp rx=0 tx=0).stop 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- 172.21.15.5:0/2639934782 shutdown_connections 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 --2- 172.21.15.5:0/2639934782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c09e8e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- 172.21.15.5:0/2639934782 >> 172.21.15.5:0/2639934782 conn(0x7f1f4c0a45d0 msgr2=0x7f1f4c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- 172.21.15.5:0/2639934782 shutdown_connections 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- 172.21.15.5:0/2639934782 wait complete. 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 10 monclient: build_initial_monmap 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 10 monclient: monmap: 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 librados: starting msgr at 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 librados: starting objecter 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 Processor -- start 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 -- start start 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 librados: setting wanted keys 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 1 librados: calling monclient init 2024-09-16T10:57:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.428+0000 7f1f50937700 10 monclient: init 2024-09-16T10:57:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:57:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-c con 0x7f1f4c071870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-b con 0x7f1f4c09aa50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f4c1376e0 0x7f1f4c120ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): picked mon.noname-a con 0x7f1f4c1376e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c124990 con 0x7f1f4c1376e0 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1234f0 con 0x7f1f4c09aa50 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c136e20 con 0x7f1f4c071870 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient(hunting): authenticate will time out at 1332.008915s 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f4c1376e0 0x7f1f4c120ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38746/0 (socket says 172.21.15.5:38746) 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2085059709 learned_addr learned my addr 172.21.15.5:0/2085059709 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): get_auth_request con 0x7f1f4c071870 auth_method 0 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): get_auth_request con 0x7f1f4c1376e0 auth_method 0 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): get_auth_request con 0x7f1f4c09aa50 auth_method 0 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient(hunting): handle_auth_done global_id 54275 payload 824 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2085059709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 msgr2=0x7f1f4c136530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c136530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f4c1376e0 msgr2=0x7f1f4c120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f4c1376e0 0x7f1f4c120ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient: _finish_hunting 0 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 monclient: found mon.noname-c 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 -- 172.21.15.5:0/2085059709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c00ab80 con 0x7f1f4c071870 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4ad9d700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c09afc0 tx=0x7f1f3c004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 client.?.objecter ms_handle_connect 0x7f1f4c071870 2024-09-16T10:57:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c008650 con 0x7f1f4c071870 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: dump: 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: _finish_auth 0 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: _check_auth_tickets 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c0087b0 con 0x7f1f4c071870 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 5 monclient: authenticate success, global_id 54275 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _renew_subs 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c0a0800 con 0x7f1f4c071870 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c008b60 con 0x7f1f4c071870 2024-09-16T10:57:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 10 monclient: dump: 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f4959a700 4 set_mon_vals no callback set 2024-09-16T10:57:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 client.54275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _renew_subs 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c09b0d0 con 0x7f1f4c071870 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f50937700 1 librados: init done 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.432+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1f3c008cc0 con 0x7f1f4c071870 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f50937700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 1 --2- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 0x7f1f2c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f50937700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f50937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c0a10f0 con 0x7f1f4c071870 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(346..346 src has 57..346) v4 ==== 5304+0+0 (secure 0 0 0) 0x7f1f3c0491b0 con 0x7f1f4c071870 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 10 client.54275.objecter ms_dispatch 0x7f1f4c000fd0 osd_map(346..346 src has 57..346) v4 2024-09-16T10:57:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 3 client.54275.objecter handle_osd_map got epochs [346,346] > 0 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 3 client.54275.objecter handle_osd_map decoding full epoch 346 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 20 client.54275.objecter dump_active .. 0 homeless 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f4a59c700 1 --2- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 0x7f1f2c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f4a59c700 10 monclient: get_auth_request con 0x7f1f2c036f60 auth_method 0 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f437fe700 10 client.54275.objecter ms_handle_connect 0x7f1f2c036f60 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.436+0000 7f1f4a59c700 1 --2- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 0x7f1f2c039420 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c1226f0 tx=0x7f1f34009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.444+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1f3c02d350 con 0x7f1f4c071870 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.444+0000 7f1f437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.444+0000 7f1f437fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:53.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:53.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:53.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1f4c0008d0 con 0x7f1f4c071870 2024-09-16T10:57:53.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f437fe700 1 -- 172.21.15.5:0/2085059709 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f1f3c012b80 con 0x7f1f4c071870 2024-09-16T10:57:53.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:53.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f437fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:53.687 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 librados: watch_flush enter 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 librados: watch_flush exit 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 20 client.54275.objecter shutdown clearing up homeless session... 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 10 client.54275.objecter successfully canceled tick 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 msgr2=0x7f1f2c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.684+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 0x7f1f2c039420 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c1226f0 tx=0x7f1f34009040 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 10 monclient: shutdown 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 msgr2=0x7f1f4c135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c09afc0 tx=0x7f1f3c004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 shutdown_connections 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1f2c036f60 0x7f1f2c039420 unknown :-1 s=CLOSED pgs=3806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1f4c1376e0 0x7f1f4c120ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f4c09aa50 0x7f1f4c136530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 --2- 172.21.15.5:0/2085059709 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1f4c071870 0x7f1f4c135ff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 >> 172.21.15.5:0/2085059709 conn(0x7f1f4c0a45d0 msgr2=0x7f1f4c070e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 shutdown_connections 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 -- 172.21.15.5:0/2085059709 wait complete. 2024-09-16T10:57:53.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.688+0000 7f1f50937700 1 librados: shutdown 2024-09-16T10:57:53.702 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:57:53.702 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:57:53.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:57:53.702 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:57:53.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 10 monclient: get_monmap_and_config 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 10 monclient: build_initial_monmap 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 10 monclient: monmap: 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:53.815482+0000 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:53.815482+0000 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:53.817 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 1 Processor -- start 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 1 -- start start 2024-09-16T10:57:53.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.812+0000 7f47d11a5700 10 monclient: init 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 0x7f47cc09ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-a con 0x7f47cc131a70 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47cc09f350 0x7f47cc131e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-c con 0x7f47cc09f350 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-b con 0x7f47cc132390 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc0a0090 con 0x7f47cc131a70 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc09b6d0 con 0x7f47cc09f350 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc09b9b0 con 0x7f47cc132390 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): authenticate will time out at 1332.392323s 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 0x7f47cc09ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38758/0 (socket says 172.21.15.5:38758) 2024-09-16T10:57:53.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 -- 172.21.15.5:0/3535125740 learned_addr learned my addr 172.21.15.5:0/3535125740 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): get_auth_request con 0x7f47cc132390 auth_method 0 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): get_auth_request con 0x7f47cc131a70 auth_method 0 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.143:3300/0 conn(0x7f47cc09f350 0x7f47cc131e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): get_auth_request con 0x7f47cc09f350 auth_method 0 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient(hunting): handle_auth_done global_id 54278 payload 274 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 -- 172.21.15.5:0/3535125740 >> v2:172.21.15.143:3300/0 conn(0x7f47cc09f350 msgr2=0x7f47cc131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.143:3300/0 conn(0x7f47cc09f350 0x7f47cc131e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 -- 172.21.15.5:0/3535125740 >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 msgr2=0x7f47cc09ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 0x7f47cc09ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient: _finish_hunting 0 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 monclient: found mon.noname-b 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 -- 172.21.15.5:0/3535125740 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc132800 con 0x7f47cc132390 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 0x7f47cc09ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cb59e700 1 --2- 172.21.15.5:0/3535125740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d150 tx=0x7f47bc00d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3289533c614abc20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 1 -- 172.21.15.5:0/3535125740 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc010070 con 0x7f47cc132390 2024-09-16T10:57:53.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: dump: 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.832 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: _finish_auth 0 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 1 -- 172.21.15.5:0/3535125740 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc00dcd0 con 0x7f47cc132390 2024-09-16T10:57:53.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 5 monclient: authenticate success, global_id 54278 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 4 set_mon_vals no callback set 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 1 -- 172.21.15.5:0/3535125740 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc00b310 con 0x7f47cc132390 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47c9d9b700 10 monclient: dump: 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.834 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: shutdown 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- 172.21.15.5:0/3535125740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 msgr2=0x7f47cc12e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- 172.21.15.5:0/3535125740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d150 tx=0x7f47bc00d460 comp rx=0 tx=0).stop 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- 172.21.15.5:0/3535125740 shutdown_connections 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- 172.21.15.5:0/3535125740 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc132390 0x7f47cc12e7c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.143:3300/0 conn(0x7f47cc09f350 0x7f47cc131e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- 172.21.15.5:0/3535125740 >> v2:172.21.15.5:3300/0 conn(0x7f47cc131a70 0x7f47cc09ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- 172.21.15.5:0/3535125740 >> 172.21.15.5:0/3535125740 conn(0x7f47cc0a0b30 msgr2=0x7f47cc0a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- 172.21.15.5:0/3535125740 shutdown_connections 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- 172.21.15.5:0/3535125740 wait complete. 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: build_initial_monmap 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: monmap: 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:53.836 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 librados: starting msgr at 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 librados: starting objecter 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 Processor -- start 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- start start 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 librados: setting wanted keys 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 librados: calling monclient init 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: init 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:53.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc09f350 0x7f47cc1c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-a con 0x7f47cc09f350 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 0x7f47cc12eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-c con 0x7f47cc131a70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47cc132390 0x7f47cc12f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): picked mon.noname-b con 0x7f47cc132390 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0a2070 con 0x7f47cc09f350 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc11d6b0 con 0x7f47cc132390 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc11f220 con 0x7f47cc131a70 2024-09-16T10:57:53.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47d11a5700 10 monclient(hunting): authenticate will time out at 1332.395835s 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 0x7f47cc12eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc09f350 0x7f47cc1c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47ca59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 0x7f47cc12eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38768/0 (socket says 172.21.15.5:38768) 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.816+0000 7f47cad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc09f350 0x7f47cc1c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60676/0 (socket says 172.21.15.5:60676) 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 -- 172.21.15.5:0/1562965840 learned_addr learned my addr 172.21.15.5:0/1562965840 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): get_auth_request con 0x7f47cc131a70 auth_method 0 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): get_auth_request con 0x7f47cc09f350 auth_method 0 2024-09-16T10:57:53.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47cc132390 0x7f47cc12f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): get_auth_request con 0x7f47cc132390 auth_method 0 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient(hunting): handle_auth_done global_id 54281 payload 824 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47cc132390 msgr2=0x7f47cc12f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47cc132390 0x7f47cc12f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc09f350 msgr2=0x7f47cc1c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc09f350 0x7f47cc1c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient: _finish_hunting 0 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 monclient: found mon.noname-c 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c00094b0 con 0x7f47cc131a70 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47ca59c700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 0x7f47cc12eb10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47c400e870 tx=0x7f47c400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.?.objecter ms_handle_connect 0x7f47cc131a70 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:53.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40095c0 con 0x7f47cc131a70 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: dump: 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.843 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: _finish_auth 0 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: _check_auth_tickets 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 5 monclient: authenticate success, global_id 54281 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c4013680 con 0x7f47cc131a70 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 monclient: _renew_subs 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc00bd40 con 0x7f47cc131a70 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4013a10 con 0x7f47cc131a70 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:53.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: dump: 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47c8d99700 4 set_mon_vals no callback set 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 client.54281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 monclient: _renew_subs 2024-09-16T10:57:53.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc11e230 con 0x7f47cc131a70 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 1 librados: init done 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 0x7f47cc02b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc067580 con 0x7f47cc11bd70 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f47c4013b70 con 0x7f47cc131a70 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 0x7f47cc02b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient: get_auth_request con 0x7f47cc11bd70 auth_method 0 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 --2- 172.21.15.5:0/1562965840 >> v2:172.21.15.143:6812/1098205034 conn(0x7f47b8036f90 0x7f47b8039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f47c4047c80 con 0x7f47cc131a70 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.54281.objecter ms_dispatch 0x7f47cc000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 3 client.54281.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 3 client.54281.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 10 monclient(hunting): handle_auth_done global_id 28905 payload 824 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47cc09ae40 con 0x7f47cc11bd70 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 20 client.54281.objecter dump_active .. 0 homeless 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 1 --2- 172.21.15.5:0/1562965840 >> v2:172.21.15.143:6812/1098205034 conn(0x7f47b8036f90 0x7f47b8039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 10 monclient: get_auth_request con 0x7f47b8036f90 auth_method 0 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cad9d700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 0x7f47cc02b6a0 secure :-1 s=READY pgs=6907 cs=0 l=1 rev1=1 crypto rx=0x7f47c0011c00 tx=0x7f47c0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.54281.objecter ms_handle_connect 0x7f47cc11bd70 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.54281.objecter resend_mon_ops 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000fb10 con 0x7f47cc11bd70 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: dump: 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.848 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47cb59e700 1 --2- 172.21.15.5:0/1562965840 >> v2:172.21.15.143:6812/1098205034 conn(0x7f47b8036f90 0x7f47b8039450 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7f47cc11df40 tx=0x7f47bc00b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 client.54281.objecter ms_handle_connect 0x7f47b8036f90 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f47c000fc70 con 0x7f47cc11bd70 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 msgr2=0x7f47cc02b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 0x7f47cc02b6a0 secure :-1 s=READY pgs=6907 cs=0 l=1 rev1=1 crypto rx=0x7f47c0011c00 tx=0x7f47c0009f60 comp rx=0 tx=0).stop 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 msgr2=0x7f47cc02b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:53.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.820+0000 7f47b37fe700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11bd70 0x7f47cc02b6a0 secure :-1 s=CLOSED pgs=6907 cs=0 l=1 rev1=1 crypto rx=0x7f47c0011c00 tx=0x7f47c0009f60 comp rx=0 tx=0).stop 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 0x7f47cc02bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc09b210 con 0x7f47cc11c150 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 0x7f47cc02bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient: get_auth_request con 0x7f47cc11c150 auth_method 0 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:53.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 10 monclient(hunting): handle_auth_done global_id 28908 payload 824 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 1 -- 172.21.15.5:0/1562965840 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47cc06f200 con 0x7f47cc11c150 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 client.54281.objecter ms_handle_connect 0x7f47cc11c150 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cad9d700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 0x7f47cc02bbe0 secure :-1 s=READY pgs=6908 cs=0 l=1 rev1=1 crypto rx=0x7f47c0011c00 tx=0x7f47c0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 client.54281.objecter resend_mon_ops 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0014850 con 0x7f47cc11c150 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:53.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 monclient: dump: 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f47c002c420 con 0x7f47cc11c150 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 msgr2=0x7f47cc02bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 0x7f47cc02bbe0 secure :-1 s=READY pgs=6908 cs=0 l=1 rev1=1 crypto rx=0x7f47c0011c00 tx=0x7f47c0004770 comp rx=0 tx=0).stop 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 msgr2=0x7f47cc02bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47b37fe700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47cc11c150 0x7f47cc02bbe0 unknown :-1 s=CLOSED pgs=6908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:53.856 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:53.857 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:53.858 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:53.859 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:53.860 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:53.861 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:53.862 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:53.863 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.864 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:53.865 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.866 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:53.867 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:53.868 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.869 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.870 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:53.871 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:53.872 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:53.872 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:53.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 librados: watch_flush enter 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 librados: watch_flush exit 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 20 client.54281.objecter shutdown clearing up homeless session... 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 client.54281.objecter successfully canceled tick 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 >> v2:172.21.15.143:6812/1098205034 conn(0x7f47b8036f90 msgr2=0x7f47b8039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 --2- 172.21.15.5:0/1562965840 >> v2:172.21.15.143:6812/1098205034 conn(0x7f47b8036f90 0x7f47b8039450 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7f47cc11df40 tx=0x7f47bc00b000 comp rx=0 tx=0).stop 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 10 monclient: shutdown 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 msgr2=0x7f47cc12eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 --2- 172.21.15.5:0/1562965840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f47cc131a70 0x7f47cc12eb10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f47c400e870 tx=0x7f47c400eb80 comp rx=0 tx=0).stop 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cb59e700 1 -- 172.21.15.5:0/1562965840 reap_dead start 2024-09-16T10:57:53.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47cb59e700 1 -- 172.21.15.5:0/1562965840 reap_dead start 2024-09-16T10:57:53.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 shutdown_connections 2024-09-16T10:57:53.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 >> 172.21.15.5:0/1562965840 conn(0x7f47cc0a0b30 msgr2=0x7f47cc130dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:53.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 shutdown_connections 2024-09-16T10:57:53.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 -- 172.21.15.5:0/1562965840 wait complete. 2024-09-16T10:57:53.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.828+0000 7f47d11a5700 1 librados: shutdown 2024-09-16T10:57:53.875 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:53.875 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:57:53.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: get_monmap_and_config 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: build_initial_monmap 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: monmap: 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:53.994 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:53.993777+0000 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:53.993777+0000 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:53.995 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:53.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 Processor -- start 2024-09-16T10:57:53.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 -- start start 2024-09-16T10:57:53.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: init 2024-09-16T10:57:53.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:53.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T10:57:53.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81b00a1dc0 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:53.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81b0072450 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f81b0070e40 0x7f81b013ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-c con 0x7f81b0070e40 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b00a21a0 con 0x7f81b0072450 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b0132c80 con 0x7f81b00a1dc0 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b006d9e0 con 0x7f81b0070e40 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.992+0000 7f81b6ae2700 10 monclient(hunting): authenticate will time out at 1332.571937s 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b52df700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f81b0072450 0x7f81b0137600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46916/0 (socket says 172.21.15.5:46916) 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b52df700 1 -- 172.21.15.5:0/4057836789 learned_addr learned my addr 172.21.15.5:0/4057836789 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 --2- 172.21.15.5:0/4057836789 >> v2:172.21.15.143:3300/0 conn(0x7f81b0070e40 0x7f81b013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): get_auth_request con 0x7f81b0070e40 auth_method 0 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 1 -- 172.21.15.5:0/4057836789 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307702758 0 0) 0x7f81b00a21a0 con 0x7f81b0072450 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient(hunting): _negotiate 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient(hunting): my global_id is 28911 2024-09-16T10:57:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 1 -- 172.21.15.5:0/4057836789 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f819c0035d0 con 0x7f81b0072450 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient(hunting): handle_auth_done global_id 64138 payload 274 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 -- 172.21.15.5:0/4057836789 >> v1:172.21.15.143:6789/0 conn(0x7f81b00a1dc0 legacy=0x7f81b0070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 -- 172.21.15.5:0/4057836789 >> v1:172.21.15.5:6789/0 conn(0x7f81b0072450 legacy=0x7f81b0137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 monclient: found mon.noname-c 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 -- 172.21.15.5:0/4057836789 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b006e960 con 0x7f81b0070e40 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b62e1700 1 --2- 172.21.15.5:0/4057836789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0070e40 0x7f81b013ad30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009d90 tx=0x7f81a800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9df09675c48c4da9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 1 -- 172.21.15.5:0/4057836789 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800e070 con 0x7f81b0070e40 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:54.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: dump: 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 5 monclient: authenticate success, global_id 64138 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 1 -- 172.21.15.5:0/4057836789 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800bb20 con 0x7f81b0070e40 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 1 -- 172.21.15.5:0/4057836789 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800be90 con 0x7f81b0070e40 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 4 set_mon_vals no callback set 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b4ade700 10 monclient: dump: 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.005 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 10 monclient: shutdown 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 -- 172.21.15.5:0/4057836789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0070e40 msgr2=0x7f81b013ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/4057836789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0070e40 0x7f81b013ad30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009d90 tx=0x7f81a800b330 comp rx=0 tx=0).stop 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 -- 172.21.15.5:0/4057836789 shutdown_connections 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/4057836789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0070e40 0x7f81b013ad30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 -- 172.21.15.5:0/4057836789 >> 172.21.15.5:0/4057836789 conn(0x7f81b00a12a0 msgr2=0x7f81b00a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 -- 172.21.15.5:0/4057836789 shutdown_connections 2024-09-16T10:57:54.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:53.996+0000 7f81b6ae2700 1 -- 172.21.15.5:0/4057836789 wait complete. 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: build_initial_monmap 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: monmap: 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 librados: starting msgr at 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 librados: starting objecter 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 Processor -- start 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 -- start start 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 librados: setting wanted keys 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 librados: calling monclient init 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: init 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0070e40 0x7f81b006c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-c con 0x7f81b0070e40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0072450 0x7f81b01c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-b con 0x7f81b0072450 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b00a1dc0 0x7f81b01307e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): picked mon.noname-a con 0x7f81b00a1dc0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00717d0 con 0x7f81b00a1dc0 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b01c93c0 con 0x7f81b0072450 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b013acd0 con 0x7f81b0070e40 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient(hunting): authenticate will time out at 1332.578599s 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0070e40 0x7f81b006c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0070e40 0x7f81b006c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38770/0 (socket says 172.21.15.5:38770) 2024-09-16T10:57:54.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 1 -- 172.21.15.5:0/644116680 learned_addr learned my addr 172.21.15.5:0/644116680 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): get_auth_request con 0x7f81b0070e40 auth_method 0 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b00a1dc0 0x7f81b01307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): get_auth_request con 0x7f81b00a1dc0 auth_method 0 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0072450 0x7f81b01c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 10 monclient(hunting): get_auth_request con 0x7f81b0072450 auth_method 0 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient(hunting): handle_auth_done global_id 28914 payload 824 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0070e40 msgr2=0x7f81b006c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0070e40 0x7f81b006c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0072450 msgr2=0x7f81b01c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0072450 0x7f81b01c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 monclient: found mon.noname-a 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8009a70 con 0x7f81b00a1dc0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b52df700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81b0072450 0x7f81b01c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b62e1700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b00a1dc0 0x7f81b01307e0 secure :-1 s=READY pgs=6910 cs=0 l=1 rev1=1 crypto rx=0x7f81a8016000 tx=0x7f81a8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 client.?.objecter ms_handle_connect 0x7f81b00a1dc0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a8004930 con 0x7f81b00a1dc0 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:57:54.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: dump: 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 5 monclient: authenticate success, global_id 28914 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a8020070 con 0x7f81b00a1dc0 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: _renew_subs 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b013afb0 con 0x7f81b00a1dc0 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80276c0 con 0x7f81b00a1dc0 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a7fff700 4 set_mon_vals no callback set 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81b6ae2700 10 client.28914.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.000+0000 7f81a67fc700 10 monclient: dump: 2024-09-16T10:57:54.020 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 10 monclient: _renew_subs 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:57:54.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b0130d20 con 0x7f81b00a1dc0 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 1 librados: init done 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 0x7f81b01389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b0074c30 con 0x7f81b01385f0 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 0x7f81b01389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient: get_auth_request con 0x7f81b01385f0 auth_method 0 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f81a8012430 con 0x7f81b00a1dc0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_done global_id 54287 payload 824 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 --2- 172.21.15.5:0/644116680 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8194036f60 0x7f8194039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81b0132930 con 0x7f81b01385f0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f81a8022070 con 0x7f81b00a1dc0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 client.28914.objecter ms_dispatch 0x7f81b0000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 3 client.28914.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 3 client.28914.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 20 client.28914.objecter dump_active .. 0 homeless 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 client.28914.objecter ms_handle_connect 0x7f81b01385f0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 client.28914.objecter resend_mon_ops 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b5ae0700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 0x7f81b01389d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00f6f0 tx=0x7f81ac00fab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b52df700 1 --2- 172.21.15.5:0/644116680 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8194036f60 0x7f8194039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b52df700 10 monclient: get_auth_request con 0x7f8194036f60 auth_method 0 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0052e0 con 0x7f81b01385f0 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 monclient: dump: 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f81ac005440 con 0x7f81b01385f0 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 msgr2=0x7f81b01389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 0x7f81b01389d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00f6f0 tx=0x7f81ac00fab0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 msgr2=0x7f81b01389d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b01385f0 0x7f81b01389d0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00f6f0 tx=0x7f81ac00fab0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81a67fc700 10 client.28914.objecter ms_handle_connect 0x7f8194036f60 2024-09-16T10:57:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.004+0000 7f81b52df700 1 --2- 172.21.15.5:0/644116680 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8194036f60 0x7f8194039420 secure :-1 s=READY pgs=3808 cs=0 l=1 rev1=1 crypto rx=0x7f81b013a310 tx=0x7f81a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 0x7f81b009ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b011afb0 con 0x7f81b0138f10 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 0x7f81b009ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient: get_auth_request con 0x7f81b0138f10 auth_method 0 2024-09-16T10:57:54.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 10 monclient(hunting): handle_auth_done global_id 54290 payload 824 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 1 -- 172.21.15.5:0/644116680 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81b00a25a0 con 0x7f81b0138f10 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b5ae0700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 0x7f81b009ca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0015a0 tx=0x7f81ac001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 client.28914.objecter ms_handle_connect 0x7f81b0138f10 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 client.28914.objecter resend_mon_ops 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac023070 con 0x7f81b0138f10 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 monclient: dump: 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f81ac003cc0 con 0x7f81b0138f10 2024-09-16T10:57:54.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 msgr2=0x7f81b009ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 0x7f81b009ca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0015a0 tx=0x7f81ac001120 comp rx=0 tx=0).stop 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 msgr2=0x7f81b009ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81a67fc700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81b0138f10 0x7f81b009ca60 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0015a0 tx=0x7f81ac001120 comp rx=0 tx=0).stop 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:54.030 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:54.031 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.032 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.033 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:54.034 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.035 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.036 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.037 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.038 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:54.039 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.041 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.042 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.043 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:54.044 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 librados: watch_flush enter 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 librados: watch_flush exit 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 20 client.28914.objecter shutdown clearing up homeless session... 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 client.28914.objecter successfully canceled tick 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8194036f60 msgr2=0x7f8194039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/644116680 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8194036f60 0x7f8194039420 secure :-1 s=READY pgs=3808 cs=0 l=1 rev1=1 crypto rx=0x7f81b013a310 tx=0x7f81a0009040 comp rx=0 tx=0).stop 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 10 monclient: shutdown 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b00a1dc0 msgr2=0x7f81b01307e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 --2- 172.21.15.5:0/644116680 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b00a1dc0 0x7f81b01307e0 secure :-1 s=READY pgs=6910 cs=0 l=1 rev1=1 crypto rx=0x7f81a8016000 tx=0x7f81a8004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b62e1700 1 -- 172.21.15.5:0/644116680 reap_dead start 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b62e1700 1 -- 172.21.15.5:0/644116680 reap_dead start 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 shutdown_connections 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 >> 172.21.15.5:0/644116680 conn(0x7f81b00a12a0 msgr2=0x7f81b00a00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 shutdown_connections 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 -- 172.21.15.5:0/644116680 wait complete. 2024-09-16T10:57:54.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.016+0000 7f81b6ae2700 1 librados: shutdown 2024-09-16T10:57:54.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:54.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:57:54.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: get_monmap_and_config 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: build_initial_monmap 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: monmap: 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:54.161621+0000 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:54.161621+0000 2024-09-16T10:57:54.162 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:54.163 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 Processor -- start 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 -- start start 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: init 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89e00a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89e00a3770 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:54.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e013e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-b con 0x7f89e013c3a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00a3db0 con 0x7f89e00a3770 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00a4b70 con 0x7f89e00a2c80 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00a0850 con 0x7f89e013c3a0 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e69b7700 10 monclient(hunting): authenticate will time out at 1332.738982s 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e51b4700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f89e00a3770 0x7f89e013ac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46926/0 (socket says 172.21.15.5:46926) 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e51b4700 1 -- 172.21.15.5:0/2474291528 learned_addr learned my addr 172.21.15.5:0/2474291528 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 1 --2- 172.21.15.5:0/2474291528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e013e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): get_auth_request con 0x7f89e013c3a0 auth_method 0 2024-09-16T10:57:54.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390066762 0 0) 0x7f89e00a3db0 con 0x7f89e00a3770 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 10 monclient(hunting): _negotiate 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 10 monclient(hunting): my global_id is 28917 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc0035d0 con 0x7f89e00a3770 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient(hunting): handle_auth_done global_id 54293 payload 274 2024-09-16T10:57:54.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625307744 0 0) 0x7f89e00a4b70 con 0x7f89e00a2c80 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 1 -- 172.21.15.5:0/2474291528 >> v1:172.21.15.143:6789/0 conn(0x7f89e00a2c80 legacy=0x7f89e00a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 1 -- 172.21.15.5:0/2474291528 >> v1:172.21.15.5:6789/0 conn(0x7f89e00a3770 legacy=0x7f89e013ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 1 monclient: found mon.noname-b 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.160+0000 7f89e61b6700 1 -- 172.21.15.5:0/2474291528 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00a4180 con 0x7f89e013c3a0 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e61b6700 1 --2- 172.21.15.5:0/2474291528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e013e790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009dc0 tx=0x7f89d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fa47027caef52dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d800e070 con 0x7f89e013c3a0 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:54.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: dump: 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 5 monclient: authenticate success, global_id 54293 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d800bb20 con 0x7f89e013c3a0 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 4 set_mon_vals no callback set 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 1 -- 172.21.15.5:0/2474291528 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d800bed0 con 0x7f89e013c3a0 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e49b3700 10 monclient: dump: 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.172 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 10 monclient: shutdown 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 -- 172.21.15.5:0/2474291528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 msgr2=0x7f89e013e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 --2- 172.21.15.5:0/2474291528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e013e790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009dc0 tx=0x7f89d800b330 comp rx=0 tx=0).stop 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 -- 172.21.15.5:0/2474291528 shutdown_connections 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 --2- 172.21.15.5:0/2474291528 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e013e790 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 -- 172.21.15.5:0/2474291528 >> 172.21.15.5:0/2474291528 conn(0x7f89e00a4d80 msgr2=0x7f89e00a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 -- 172.21.15.5:0/2474291528 shutdown_connections 2024-09-16T10:57:54.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 -- 172.21.15.5:0/2474291528 wait complete. 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 10 monclient: build_initial_monmap 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 10 monclient: monmap: 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:54.180 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 librados: starting msgr at 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 librados: starting objecter 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.164+0000 7f89e69b7700 1 Processor -- start 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- start start 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 librados: setting wanted keys 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 librados: calling monclient init 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: init 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e00a2c80 0x7f89e01376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-b con 0x7f89e00a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:54.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e00a3770 0x7f89e0137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-a con 0x7f89e00a3770 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): picked mon.noname-c con 0x7f89e013c3a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00a74b0 con 0x7f89e00a3770 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0122ee0 con 0x7f89e00a2c80 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0121ca0 con 0x7f89e013c3a0 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): authenticate will time out at 1332.744728s 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e00a3770 0x7f89e0137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38808/0 (socket says 172.21.15.5:38808) 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 -- 172.21.15.5:0/3570895644 learned_addr learned my addr 172.21.15.5:0/3570895644 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): get_auth_request con 0x7f89e013c3a0 auth_method 0 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): get_auth_request con 0x7f89e00a3770 auth_method 0 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e00a2c80 0x7f89e01376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e51b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): get_auth_request con 0x7f89e00a2c80 auth_method 0 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient(hunting): handle_auth_done global_id 54296 payload 824 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e00a2c80 msgr2=0x7f89e01376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e00a2c80 0x7f89e01376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e00a3770 msgr2=0x7f89e0137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f89e00a3770 0x7f89e0137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 monclient: found mon.noname-c 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d8009a70 con 0x7f89e013c3a0 2024-09-16T10:57:54.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e61b6700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009f30 tx=0x7f89d8004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 client.?.objecter ms_handle_connect 0x7f89e013c3a0 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d801b810 con 0x7f89e013c3a0 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: dump: 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.186 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 5 monclient: authenticate success, global_id 54296 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d8020070 con 0x7f89e013c3a0 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _renew_subs 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e01234a0 con 0x7f89e013c3a0 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d801bbb0 con 0x7f89e013c3a0 2024-09-16T10:57:54.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 client.54296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d7fff700 4 set_mon_vals no callback set 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 10 monclient: dump: 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.188 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _renew_subs 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e01231c0 con 0x7f89e013c3a0 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 librados: init done 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f89d801bd10 con 0x7f89e013c3a0 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 0x7f89e002b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0067380 con 0x7f89e0120230 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89d67fc700 1 --2- 172.21.15.5:0/3570895644 >> v2:172.21.15.143:6812/1098205034 conn(0x7f89c4036f60 0x7f89c4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.168+0000 7f89e59b5700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 0x7f89e002b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f89d8022070 con 0x7f89e013c3a0 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient: get_auth_request con 0x7f89e0120230 auth_method 0 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 client.54296.objecter ms_dispatch 0x7f89e0000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 3 client.54296.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 3 client.54296.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 20 client.54296.objecter dump_active .. 0 homeless 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e51b4700 1 --2- 172.21.15.5:0/3570895644 >> v2:172.21.15.143:6812/1098205034 conn(0x7f89c4036f60 0x7f89c4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e51b4700 10 monclient: get_auth_request con 0x7f89c4036f60 auth_method 0 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 10 monclient(hunting): handle_auth_done global_id 64153 payload 824 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89e004ec20 con 0x7f89e0120230 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e59b5700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 0x7f89e002b6a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00f770 tx=0x7f89dc00fb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 client.54296.objecter ms_handle_connect 0x7f89e0120230 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 client.54296.objecter resend_mon_ops 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 client.54296.objecter ms_handle_connect 0x7f89c4036f60 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89e51b4700 1 --2- 172.21.15.5:0/3570895644 >> v2:172.21.15.143:6812/1098205034 conn(0x7f89c4036f60 0x7f89c4039420 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f89e00a3710 tx=0x7f89d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0052e0 con 0x7f89e0120230 2024-09-16T10:57:54.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 monclient: dump: 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f89dc005440 con 0x7f89e0120230 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 msgr2=0x7f89e002b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 0x7f89e002b6a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00f770 tx=0x7f89dc00fb30 comp rx=0 tx=0).stop 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 msgr2=0x7f89e002b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:54.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.172+0000 7f89d67fc700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120230 0x7f89e002b6a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e69b7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:57:54.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e69b7700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 0x7f89e002bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e69b7700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e009fa10 con 0x7f89e0120610 2024-09-16T10:57:54.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 0x7f89e002bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient: get_auth_request con 0x7f89e0120610 auth_method 0 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 10 monclient(hunting): handle_auth_done global_id 64156 payload 824 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 1 -- 172.21.15.5:0/3570895644 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f89e00a4400 con 0x7f89e0120610 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89e59b5700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 0x7f89e002bbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f89dc017000 tx=0x7f89dc007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 client.54296.objecter ms_handle_connect 0x7f89e0120610 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 client.54296.objecter resend_mon_ops 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc023070 con 0x7f89e0120610 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 monclient: dump: 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4611 (secure 0 0 0) 0x7f89dc027410 con 0x7f89e0120610 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.184+0000 7f89d67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 msgr2=0x7f89e002bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89d67fc700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 0x7f89e002bbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f89dc017000 tx=0x7f89dc007f90 comp rx=0 tx=0).stop 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89d67fc700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 msgr2=0x7f89e002bbe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89d67fc700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89e0120610 0x7f89e002bbe0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 36, 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.200 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:57:54.202 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:57:54.204 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.208 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.211 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 10 librados: watch_flush enter 2024-09-16T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 10 librados: watch_flush exit 2024-09-16T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 20 client.54296.objecter shutdown clearing up homeless session... 2024-09-16T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 10 client.54296.objecter successfully canceled tick 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 >> v2:172.21.15.143:6812/1098205034 conn(0x7f89c4036f60 msgr2=0x7f89c4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 --2- 172.21.15.5:0/3570895644 >> v2:172.21.15.143:6812/1098205034 conn(0x7f89c4036f60 0x7f89c4039420 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f89e00a3710 tx=0x7f89d0009040 comp rx=0 tx=0).stop 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 10 monclient: shutdown 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 msgr2=0x7f89e011f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e61b6700 1 -- 172.21.15.5:0/3570895644 reap_dead start 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009f30 tx=0x7f89d8004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 shutdown_connections 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 --2- 172.21.15.5:0/3570895644 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f89e013c3a0 0x7f89e011f520 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 >> 172.21.15.5:0/3570895644 conn(0x7f89e00a4d80 msgr2=0x7f89e013aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 shutdown_connections 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 -- 172.21.15.5:0/3570895644 wait complete. 2024-09-16T10:57:54.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.188+0000 7f89e69b7700 1 librados: shutdown 2024-09-16T10:57:54.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:57:54.218 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T10:57:54.218 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:57:54.218 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T10:57:54.218 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T10:57:54.218 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T10:57:54.219 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:57:54.261 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T10:57:54.262 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:57:54.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:54.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 10 monclient: get_monmap_and_config 2024-09-16T10:57:54.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 10 monclient: build_initial_monmap 2024-09-16T10:57:54.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 10 monclient: monmap: 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:54.379866+0000 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:54.379866+0000 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:54.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 1 Processor -- start 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 1 -- start start 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.376+0000 7f61e158b700 10 monclient: init 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T10:57:54.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 0x7f61dc09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-a con 0x7f61dc09e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61dc09d6a0 0x7f61dc09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-c con 0x7f61dc09d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61dc09dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc070e40 con 0x7f61dc09dfd0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc0a01d0 con 0x7f61dc09e8e0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc09f740 con 0x7f61dc09d6a0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient(hunting): authenticate will time out at 1332.956257s 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61dbfff700 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 msgr2=0x7f61dc09ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:57:54.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e0d8a700 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f61dc09dfd0 legacy=0x7f61dc1336c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61dbfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 0x7f61dc09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61dc09d6a0 0x7f61dc09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61dc09d6a0 0x7f61dc09f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:50130/0 (socket says 172.21.15.5:50130) 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 -- 172.21.15.5:0/3192441778 learned_addr learned my addr 172.21.15.5:0/3192441778 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): get_auth_request con 0x7f61dc09d6a0 auth_method 0 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient(hunting): handle_auth_done global_id 64159 payload 274 2024-09-16T10:57:54.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 -- 172.21.15.5:0/3192441778 >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 msgr2=0x7f61dc09ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 --2- 172.21.15.5:0/3192441778 >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 0x7f61dc09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 -- 172.21.15.5:0/3192441778 >> v1:172.21.15.5:6789/0 conn(0x7f61dc09dfd0 legacy=0x7f61dc1336c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 monclient: found mon.noname-c 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 -- 172.21.15.5:0/3192441778 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc09fef0 con 0x7f61dc09d6a0 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61db7fe700 1 --2- 172.21.15.5:0/3192441778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09d6a0 0x7f61dc09f200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009910 tx=0x7f61cc009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b9c955d1bb991b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 1 -- 172.21.15.5:0/3192441778 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc004230 con 0x7f61dc09d6a0 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:54.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: dump: 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 5 monclient: authenticate success, global_id 64159 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 1 -- 172.21.15.5:0/3192441778 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc004390 con 0x7f61dc09d6a0 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 1 -- 172.21.15.5:0/3192441778 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc009e60 con 0x7f61dc09d6a0 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61daffd700 10 monclient: dump: 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.390 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 4 set_mon_vals no callback set 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 10 monclient: shutdown 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- 172.21.15.5:0/3192441778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09d6a0 msgr2=0x7f61dc09f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 --2- 172.21.15.5:0/3192441778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09d6a0 0x7f61dc09f200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009910 tx=0x7f61cc009c20 comp rx=0 tx=0).stop 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- 172.21.15.5:0/3192441778 shutdown_connections 2024-09-16T10:57:54.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 --2- 172.21.15.5:0/3192441778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09d6a0 0x7f61dc09f200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 --2- 172.21.15.5:0/3192441778 >> v2:172.21.15.5:3300/0 conn(0x7f61dc09e8e0 0x7f61dc09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- 172.21.15.5:0/3192441778 >> 172.21.15.5:0/3192441778 conn(0x7f61dc0a45d0 msgr2=0x7f61dc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- 172.21.15.5:0/3192441778 shutdown_connections 2024-09-16T10:57:54.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.380+0000 7f61e158b700 1 -- 172.21.15.5:0/3192441778 wait complete. 2024-09-16T10:57:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: build_initial_monmap 2024-09-16T10:57:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: monmap: 2024-09-16T10:57:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:54.397 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 librados: starting msgr at 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 librados: starting objecter 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 Processor -- start 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- start start 2024-09-16T10:57:54.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 librados: setting wanted keys 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 librados: calling monclient init 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: init 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-c con 0x7f61dc09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 0x7f61dc132d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-a con 0x7f61dc09dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 0x7f61dc133270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): picked mon.noname-b con 0x7f61dc09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc11eda0 con 0x7f61dc09dfd0 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1c8bd0 con 0x7f61dc09e8e0 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1c9340 con 0x7f61dc09d6a0 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient(hunting): authenticate will time out at 1332.961637s 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61db7fe700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 msgr2=0x7f61dc132d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61db7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 0x7f61dc132d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38834/0 (socket says 172.21.15.5:38834) 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 -- 172.21.15.5:0/1976747767 learned_addr learned my addr 172.21.15.5:0/1976747767 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): get_auth_request con 0x7f61dc09d6a0 auth_method 0 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e0d8a700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 0x7f61dc133270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient(hunting): handle_auth_done global_id 54299 payload 824 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 -- 172.21.15.5:0/1976747767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 msgr2=0x7f61dc133270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 0x7f61dc133270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 -- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 msgr2=0x7f61dc132d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 0x7f61dc132d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient: _finish_hunting 0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 monclient: found mon.noname-c 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 -- 172.21.15.5:0/1976747767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc009570 con 0x7f61dc09d6a0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e0d8a700 10 monclient: get_auth_request con 0x7f61dc09e8e0 auth_method 0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e0d8a700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 0x7f61dc133270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 client.?.objecter ms_handle_connect 0x7f61dc09d6a0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61dbfff700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b5a0 tx=0x7f61d000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0003e70 con 0x7f61dc09d6a0 2024-09-16T10:57:54.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: dump: 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: _finish_auth 0 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d0005070 con 0x7f61dc09d6a0 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0005440 con 0x7f61dc09d6a0 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 5 monclient: authenticate success, global_id 54299 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _renew_subs 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61da7fc700 4 set_mon_vals no callback set 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc070150 con 0x7f61dc09d6a0 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:54.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61d8ff9700 10 monclient: dump: 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 client.54299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _renew_subs 2024-09-16T10:57:54.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc11e160 con 0x7f61dc09d6a0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.384+0000 7f61e158b700 1 librados: init done 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61e158b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61e158b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61e158b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc11eb10 con 0x7f61dc09d6a0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f61d00055a0 con 0x7f61dc09d6a0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 1 --2- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 0x7f61c80393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f61d0047250 con 0x7f61dc09d6a0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 10 client.54299.objecter ms_dispatch 0x7f61dc000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 3 client.54299.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 3 client.54299.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:57:54.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 20 client.54299.objecter dump_active .. 0 homeless 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61db7fe700 1 --2- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 0x7f61c80393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61db7fe700 10 monclient: get_auth_request con 0x7f61c8036f10 auth_method 0 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61d8ff9700 10 client.54299.objecter ms_handle_connect 0x7f61c8036f10 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.388+0000 7f61db7fe700 1 --2- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 0x7f61c80393d0 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009910 tx=0x7f61cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.392+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f61d0018d20 con 0x7f61dc09d6a0 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.392+0000 7f61d8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:54.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.392+0000 7f61d8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:54.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61e158b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:54.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61e158b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:54.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61e158b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:54.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f61dc071370 con 0x7f61dc09d6a0 2024-09-16T10:57:54.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61d8ff9700 1 -- 172.21.15.5:0/1976747767 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f61d0016020 con 0x7f61dc09d6a0 2024-09-16T10:57:54.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61d8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:54.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.624+0000 7f61d8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:54.629 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:54.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 10 librados: watch_flush enter 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 10 librados: watch_flush exit 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 20 client.54299.objecter shutdown clearing up homeless session... 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 10 client.54299.objecter successfully canceled tick 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 msgr2=0x7f61c80393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 0x7f61c80393d0 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009910 tx=0x7f61cc008030 comp rx=0 tx=0).stop 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 10 monclient: shutdown 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 msgr2=0x7f61dc1c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b5a0 tx=0x7f61d000b960 comp rx=0 tx=0).stop 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 shutdown_connections 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61dc09e8e0 0x7f61dc133270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61c8036f10 0x7f61c80393d0 unknown :-1 s=CLOSED pgs=3810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61dc09dfd0 0x7f61dc132d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 --2- 172.21.15.5:0/1976747767 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61dc09d6a0 0x7f61dc1c8690 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 >> 172.21.15.5:0/1976747767 conn(0x7f61dc0a45d0 msgr2=0x7f61dc1bbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 shutdown_connections 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 -- 172.21.15.5:0/1976747767 wait complete. 2024-09-16T10:57:54.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:54.628+0000 7f61e158b700 1 librados: shutdown 2024-09-16T10:57:57.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:57:57.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.760+0000 7f27cb257700 10 monclient: get_monmap_and_config 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.760+0000 7f27cb257700 10 monclient: build_initial_monmap 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.760+0000 7f27cb257700 10 monclient: monmap: 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:57:57.764127+0000 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:57:57.764127+0000 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:57:57.765 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:57:57.766 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:57:57.766 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.760+0000 7f27cb257700 1 Processor -- start 2024-09-16T10:57:57.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- start start 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient: init 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27c409aa50 addr v1:172.21.15.5:6789/0 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27c4071870 addr v1:172.21.15.143:6789/0 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-b con 0x7f27c41386f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c409caa0 con 0x7f27c409aa50 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c409e3a0 con 0x7f27c4071870 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c409ae30 con 0x7f27c41386f0 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:57.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient(hunting): authenticate will time out at 1336.340644s 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27ca255700 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f27c409aa50 legacy=0x7f27c409e8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38846/0 (socket says 172.21.15.5:38846) 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 -- 172.21.15.5:0/2346445374 learned_addr learned my addr 172.21.15.5:0/2346445374 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): get_auth_request con 0x7f27c41386f0 auth_method 0 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:57.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient(hunting): handle_auth_done global_id 54302 payload 274 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 -- 172.21.15.5:0/2346445374 >> v1:172.21.15.143:6789/0 conn(0x7f27c4071870 legacy=0x7f27c409eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 -- 172.21.15.5:0/2346445374 >> v1:172.21.15.5:6789/0 conn(0x7f27c409aa50 legacy=0x7f27c409e8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient: _finish_hunting 0 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 monclient: found mon.noname-b 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 -- 172.21.15.5:0/2346445374 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c409f700 con 0x7f27c41386f0 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27caa56700 1 --2- 172.21.15.5:0/2346445374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009e80 tx=0x7f27bc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24957be9919ef6d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 1 -- 172.21.15.5:0/2346445374 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc003a40 con 0x7f27c41386f0 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: dump: 2024-09-16T10:57:57.773 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: _finish_auth 0 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: _check_auth_tickets 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 5 monclient: authenticate success, global_id 54302 2024-09-16T10:57:57.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 1 -- 172.21.15.5:0/2346445374 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc003ba0 con 0x7f27c41386f0 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient: get_monmap_and_config success 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 4 set_mon_vals no callback set 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 1 -- 172.21.15.5:0/2346445374 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00b980 con 0x7f27c41386f0 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27c9253700 10 monclient: dump: 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:57.775 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 10 monclient: shutdown 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- 172.21.15.5:0/2346445374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 msgr2=0x7f27c413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 --2- 172.21.15.5:0/2346445374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009e80 tx=0x7f27bc00b330 comp rx=0 tx=0).stop 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- 172.21.15.5:0/2346445374 shutdown_connections 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 --2- 172.21.15.5:0/2346445374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c413aae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- 172.21.15.5:0/2346445374 >> 172.21.15.5:0/2346445374 conn(0x7f27c40a45d0 msgr2=0x7f27c40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- 172.21.15.5:0/2346445374 shutdown_connections 2024-09-16T10:57:57.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.764+0000 7f27cb257700 1 -- 172.21.15.5:0/2346445374 wait complete. 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: build_initial_monmap 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: monmap: 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:57:57.777 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:57:57.780 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:57:57.780 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:57:57.780 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 librados: starting msgr at 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 librados: starting objecter 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 Processor -- start 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 -- start start 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 librados: setting wanted keys 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 librados: calling monclient init 2024-09-16T10:57:57.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: init 2024-09-16T10:57:57.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: _reopen_session rank -1 2024-09-16T10:57:57.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:57:57.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 0x7f27c41c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:57.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-b con 0x7f27c4071870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:57:57.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 0x7f27c4137c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-a con 0x7f27c409aa50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): picked mon.noname-c con 0x7f27c41386f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c411ffe0 con 0x7f27c409aa50 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c41c8940 con 0x7f27c4071870 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): start opening mon connection 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c40a3300 con 0x7f27c41386f0 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): _renew_subs 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient(hunting): authenticate will time out at 1336.346646s 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27c9a54700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 msgr2=0x7f27c4137c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:57:57.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27c9a54700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 0x7f27c4137c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38856/0 (socket says 172.21.15.5:38856) 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 -- 172.21.15.5:0/3221731023 learned_addr learned my addr 172.21.15.5:0/3221731023 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): get_auth_request con 0x7f27c41386f0 auth_method 0 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 0x7f27c41c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:57.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 10 monclient(hunting): get_auth_request con 0x7f27c4071870 auth_method 0 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient(hunting): handle_auth_done global_id 54305 payload 824 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 -- 172.21.15.5:0/3221731023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 msgr2=0x7f27c41c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 0x7f27c41c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 -- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 msgr2=0x7f27c4137c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 0x7f27c4137c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient: _finish_hunting 0 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 monclient: found mon.noname-c 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:57:57.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 -- 172.21.15.5:0/3221731023 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009b30 con 0x7f27c41386f0 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27ca255700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 0x7f27c41c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 client.?.objecter ms_handle_connect 0x7f27c41386f0 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27caa56700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00b730 tx=0x7f27bc004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc009300 con 0x7f27c41386f0 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 monclient: dump: 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 monclient: _finish_auth 0 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00de20 con 0x7f27c41386f0 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 5 monclient: authenticate success, global_id 54305 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.768+0000 7f27cb257700 10 monclient: _renew_subs 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c40a2e50 con 0x7f27c41386f0 2024-09-16T10:57:57.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00bbc0 con 0x7f27c41386f0 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 monclient: dump: 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:57:57.792 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27c8a52700 4 set_mon_vals no callback set 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 client.54305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 monclient: _renew_subs 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c411fd00 con 0x7f27c41386f0 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 1 librados: init done 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c4120120 con 0x7f27c41386f0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f27bc00bd20 con 0x7f27c41386f0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 1 --2- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 0x7f27a80393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f27bc049950 con 0x7f27c41386f0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 client.54305.objecter ms_dispatch 0x7f27c4000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 3 client.54305.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 3 client.54305.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 20 client.54305.objecter dump_active .. 0 homeless 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27ca255700 1 --2- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 0x7f27a80393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27ca255700 10 monclient: get_auth_request con 0x7f27a8036f10 auth_method 0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27ca255700 1 --2- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 0x7f27a80393d0 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f27b8003a30 tx=0x7f27b8003ae0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:57:57.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.772+0000 7f27b6ffd700 10 client.54305.objecter ms_handle_connect 0x7f27a8036f10 2024-09-16T10:57:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.780+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f27c4120120 con 0x7f27c41386f0 2024-09-16T10:57:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.780+0000 7f27b6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:57:57.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:57.780+0000 7f27b6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:57:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:57:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:57:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f27c409caa0 con 0x7f27c41386f0 2024-09-16T10:57:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27b6ffd700 1 -- 172.21.15.5:0/3221731023 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f27c409caa0 con 0x7f27c41386f0 2024-09-16T10:57:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27b6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:57:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27b6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:57:58.019 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 librados: watch_flush enter 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 librados: watch_flush exit 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 20 client.54305.objecter shutdown clearing up homeless session... 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 client.54305.objecter successfully canceled tick 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 msgr2=0x7f27a80393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 0x7f27a80393d0 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f27b8003a30 tx=0x7f27b8003ae0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 10 monclient: shutdown 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 msgr2=0x7f27c411c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00b730 tx=0x7f27bc004af0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 shutdown_connections 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f27c41386f0 0x7f27c411c670 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> v2:172.21.15.143:6812/1098205034 conn(0x7f27a8036f10 0x7f27a80393d0 unknown :-1 s=CLOSED pgs=3811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27c409aa50 0x7f27c4137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 --2- 172.21.15.5:0/3221731023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c4071870 0x7f27c41c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 >> 172.21.15.5:0/3221731023 conn(0x7f27c40a45d0 msgr2=0x7f27c4139150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 shutdown_connections 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.016+0000 7f27cb257700 1 -- 172.21.15.5:0/3221731023 wait complete. 2024-09-16T10:57:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:57:58.020+0000 7f27cb257700 1 librados: shutdown 2024-09-16T10:58:01.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:58:01.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.152+0000 7fb14881f700 10 monclient: get_monmap_and_config 2024-09-16T10:58:01.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.152+0000 7fb14881f700 10 monclient: build_initial_monmap 2024-09-16T10:58:01.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.152+0000 7fb14881f700 10 monclient: monmap: 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:58:01.156009+0000 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:58:01.156009+0000 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:58:01.157 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 Processor -- start 2024-09-16T10:58:01.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 -- start start 2024-09-16T10:58:01.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient: init 2024-09-16T10:58:01.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:01.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T10:58:01.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb144070860 0x7fb144132940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:01.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-c con 0x7fb144070860 addr v2:172.21.15.143:3300/0 2024-09-16T10:58:01.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb144070c40 addr v1:172.21.15.5:6789/0 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1440a0a40 addr v1:172.21.15.143:6789/0 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14406f3c0 con 0x7fb144070c40 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14409ef50 con 0x7fb1440a0a40 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb14406dfd0 con 0x7fb144070860 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient(hunting): authenticate will time out at 1339.734088s 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb141d9b700 1 -- >> v1:172.21.15.5:6789/0 conn(0x7fb144070c40 legacy=0x7fb1440a0330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb144070860 0x7fb144132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb142d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb1440a0a40 0x7fb14413a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:47212/0 (socket says 172.21.15.5:47212) 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb142d9d700 1 -- 172.21.15.5:0/2446935715 learned_addr learned my addr 172.21.15.5:0/2446935715 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:58:01.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): get_auth_request con 0x7fb144070860 auth_method 0 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient(hunting): handle_auth_done global_id 64168 payload 274 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 -- 172.21.15.5:0/2446935715 >> v1:172.21.15.143:6789/0 conn(0x7fb1440a0a40 legacy=0x7fb14413a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 -- 172.21.15.5:0/2446935715 >> v1:172.21.15.5:6789/0 conn(0x7fb144070c40 legacy=0x7fb1440a0330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient: _finish_hunting 0 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 monclient: found mon.noname-c 2024-09-16T10:58:01.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 -- 172.21.15.5:0/2446935715 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144074600 con 0x7fb144070860 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14259c700 1 --2- 172.21.15.5:0/2446935715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070860 0x7fb144132940 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb134009850 tx=0x7fb134009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74bb1d097df4013a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 1 -- 172.21.15.5:0/2446935715 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134004230 con 0x7fb144070860 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: dump: 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:01.164 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: _finish_auth 0 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: _check_auth_tickets 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 5 monclient: authenticate success, global_id 64168 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 1 -- 172.21.15.5:0/2446935715 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb134004390 con 0x7fb144070860 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 10 monclient: get_monmap_and_config success 2024-09-16T10:58:01.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14881f700 4 set_mon_vals no callback set 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 1 -- 172.21.15.5:0/2446935715 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1340036c0 con 0x7fb144070860 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.156+0000 7fb14159a700 10 monclient: dump: 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:01.166 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:01.167 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:01.167 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 10 monclient: shutdown 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- 172.21.15.5:0/2446935715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070860 msgr2=0x7fb144132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 --2- 172.21.15.5:0/2446935715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070860 0x7fb144132940 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb134009850 tx=0x7fb134009b60 comp rx=0 tx=0).stop 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- 172.21.15.5:0/2446935715 shutdown_connections 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 --2- 172.21.15.5:0/2446935715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070860 0x7fb144132940 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- 172.21.15.5:0/2446935715 >> 172.21.15.5:0/2446935715 conn(0x7fb1440a1a50 msgr2=0x7fb144073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- 172.21.15.5:0/2446935715 shutdown_connections 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- 172.21.15.5:0/2446935715 wait complete. 2024-09-16T10:58:01.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 10 monclient: build_initial_monmap 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 10 monclient: monmap: 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 librados: starting msgr at 2024-09-16T10:58:01.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 librados: starting objecter 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 Processor -- start 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 -- start start 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 librados: setting wanted keys 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 1 librados: calling monclient init 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.160+0000 7fb14881f700 10 monclient: init 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-c con 0x7fb144070860 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070c40 0x7fb144130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-b con 0x7fb144070c40 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:58:01.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 0x7fb144130fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): picked mon.noname-a con 0x7fb1440a0a40 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14413baf0 con 0x7fb1440a0a40 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144071990 con 0x7fb144070c40 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14413cd70 con 0x7fb144070860 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient(hunting): authenticate will time out at 1339.740361s 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb142d9d700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 msgr2=0x7fb144130fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb142d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 0x7fb144130fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:01.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:38868/0 (socket says 172.21.15.5:38868) 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 -- 172.21.15.5:0/3891022098 learned_addr learned my addr 172.21.15.5:0/3891022098 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): get_auth_request con 0x7fb144070860 auth_method 0 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070c40 0x7fb144130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): get_auth_request con 0x7fb144070c40 auth_method 0 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:01.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb141d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient(hunting): handle_auth_done global_id 54308 payload 824 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 -- 172.21.15.5:0/3891022098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070c40 msgr2=0x7fb144130a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070c40 0x7fb144130a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 -- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 msgr2=0x7fb144130fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 0x7fb144130fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient: _finish_hunting 0 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 monclient: found mon.noname-c 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:01.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 -- 172.21.15.5:0/3891022098 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340094b0 con 0x7fb144070860 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 client.?.objecter ms_handle_connect 0x7fb144070860 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14259c700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb14409f0e0 tx=0x7fb134004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134006ea0 con 0x7fb144070860 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: dump: 2024-09-16T10:58:01.178 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: _finish_auth 0 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:01.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb134011830 con 0x7fb144070860 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 5 monclient: authenticate success, global_id 54308 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _renew_subs 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14413bec0 con 0x7fb144070860 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134004610 con 0x7fb144070860 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 monclient: dump: 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:01.180 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 client.54308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _renew_subs 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb144132030 con 0x7fb144070860 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 librados: init done 2024-09-16T10:58:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb140d99700 4 set_mon_vals no callback set 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14409dfe0 con 0x7fb144070860 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb134021070 con 0x7fb144070860 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 --2- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 0x7fb1240393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7fb134049580 con 0x7fb144070860 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 10 client.54308.objecter ms_dispatch 0x7fb144000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 3 client.54308.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 3 client.54308.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.164+0000 7fb132ffd700 20 client.54308.objecter dump_active .. 0 homeless 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.168+0000 7fb141d9b700 1 --2- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 0x7fb1240393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:01.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.168+0000 7fb141d9b700 10 monclient: get_auth_request con 0x7fb124036f10 auth_method 0 2024-09-16T10:58:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.168+0000 7fb132ffd700 10 client.54308.objecter ms_handle_connect 0x7fb124036f10 2024-09-16T10:58:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.168+0000 7fb141d9b700 1 --2- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 0x7fb1240393d0 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7fb138003a30 tx=0x7fb1380091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.176+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb14409dfe0 con 0x7fb144070860 2024-09-16T10:58:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.176+0000 7fb132ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:01.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.176+0000 7fb132ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:58:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb14881f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:58:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb14881f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:58:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb14881f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb14409dfe0 con 0x7fb144070860 2024-09-16T10:58:01.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb132ffd700 1 -- 172.21.15.5:0/3891022098 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fb14409dfe0 con 0x7fb144070860 2024-09-16T10:58:01.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb132ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:58:01.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.408+0000 7fb132ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:58:01.413 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 10 librados: watch_flush enter 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 10 librados: watch_flush exit 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 20 client.54308.objecter shutdown clearing up homeless session... 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 10 client.54308.objecter successfully canceled tick 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 msgr2=0x7fb1240393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:01.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 0x7fb1240393d0 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7fb138003a30 tx=0x7fb1380091c0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 10 monclient: shutdown 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 msgr2=0x7fb14413b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb14409f0e0 tx=0x7fb134004770 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 shutdown_connections 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb124036f10 0x7fb1240393d0 unknown :-1 s=CLOSED pgs=3812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1440a0a40 0x7fb144130fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144070c40 0x7fb144130a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 --2- 172.21.15.5:0/3891022098 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb144070860 0x7fb14413b020 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 >> 172.21.15.5:0/3891022098 conn(0x7fb1440a1a50 msgr2=0x7fb144132680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 shutdown_connections 2024-09-16T10:58:01.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 -- 172.21.15.5:0/3891022098 wait complete. 2024-09-16T10:58:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:01.412+0000 7fb14881f700 1 librados: shutdown 2024-09-16T10:58:04.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: get_monmap_and_config 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: build_initial_monmap 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: monmap: 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:58:04.550460+0000 2024-09-16T10:58:04.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:58:04.550460+0000 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:58:04.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 Processor -- start 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 -- start start 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: init 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-b con 0x7f2c6409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:04.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-c con 0x7f2c6409f200 addr v2:172.21.15.143:3300/0 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c6413a460 addr v1:172.21.15.5:6789/0 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6409fb80 con 0x7f2c6413a460 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c6409f200 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409b970 con 0x7f2c6409e8e0 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c68885700 10 monclient(hunting): authenticate will time out at 1343.127060s 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c63fff700 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f2c6413a460 legacy=0x7f2c641411e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:04.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52952/0 (socket says 172.21.15.5:52952) 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 learned_addr learned my addr 172.21.15.5:0/944812328 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c6409e8e0 auth_method 0 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c6409f200 auth_method 0 2024-09-16T10:58:04.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:04.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.547+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:04.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c6413a460 legacy=0x7f2c641411e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c6409e8e0 auth_method 0 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c6409f200 auth_method 0 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:04.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:04.751+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c6413a460 legacy=0x7f2c641411e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:05.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c6409e8e0 auth_method 0 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c6409f200 auth_method 0 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:05.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:05.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.155+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:05.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c6413a460 legacy=0x7f2c641411e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:05.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c6409e8e0 auth_method 0 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c6409f200 auth_method 0 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:05.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:05.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:05.959+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): tick 2024-09-16T10:58:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:58:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 monclient(hunting): continuing hunt 2024-09-16T10:58:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:58:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 msgr2=0x7f2c6409ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c6409e8e0 0x7f2c6409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 msgr2=0x7f2c64139f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c6409f200 0x7f2c64139f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c6413a460 legacy=0x7f2c641411e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient: _add_conns ranks=[3,1] 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-a con 0x7f2c500036f0 addr v2:172.21.15.5:3300/0 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c50006260 addr v1:172.21.15.143:6789/0 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6409bc50 con 0x7f2c50006260 2024-09-16T10:58:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c500036f0 2024-09-16T10:58:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c617fa700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.547+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.751+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:07.751+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:08.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:08.155+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:08.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:08.155+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:08.959+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:08.959+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:10.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): tick 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 monclient(hunting): continuing hunt 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c50006260 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 reap_dead start 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-a con 0x7f2c500036f0 addr v2:172.21.15.5:3300/0 2024-09-16T10:58:10.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c5000a0f0 addr v1:172.21.15.5:6789/0 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-b con 0x7f2c5000a570 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6409fb80 con 0x7f2c5000a0f0 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c500036f0 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c5000a0f0 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409b970 con 0x7f2c5000a570 2024-09-16T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.547+0000 7f2c617fa700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:10.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:10.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.751+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:10.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.751+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c5000a0f0 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:10.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.751+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:10.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:10.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:10.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.155+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.155+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c5000a0f0 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:11.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.155+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:11.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.959+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.959+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c5000a0f0 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:11.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.959+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:11.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:11.963+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:13.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): tick 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 monclient(hunting): continuing hunt 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 msgr2=0x7f2c50005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.5:3300/0 conn(0x7f2c500036f0 0x7f2c50005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.5:6789/0 conn(0x7f2c5000a0f0 legacy=0x7f2c50008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient: _add_conns ranks=[4,1] 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-c con 0x7f2c5000d260 addr v2:172.21.15.143:3300/0 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c5000dd40 addr v1:172.21.15.143:6789/0 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5000e370 con 0x7f2c5000dd40 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c5000d260 2024-09-16T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c617fa700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.551+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:13.755+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:14.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:14.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:14.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:14.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.159+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:14.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:58:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:14.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:14.963+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T10:58:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:15.751+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 legacy=0x7f2c50012640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T10:58:15.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:15.755+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 legacy=0x7f2c50012640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:58:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:15.755+0000 7f2c62ffd700 1 --1- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 0x7f2c50012640 :-1 s=OPENED pgs=56 cs=1 l=1).handle_message read tag failed 2024-09-16T10:58:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:15.755+0000 7f2c62ffd700 1 --1- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 0x7f2c50012640 :-1 s=OPENED pgs=56 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:58:15.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:15.755+0000 7f2c627fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6789/0 2024-09-16T10:58:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): tick 2024-09-16T10:58:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): _check_auth_tickets 2024-09-16T10:58:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 monclient(hunting): continuing hunt 2024-09-16T10:58:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T10:58:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 legacy=0x7f2c50012640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:58:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T10:58:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 reap_dead start 2024-09-16T10:58:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c5000dd40 addr v1:172.21.15.143:6789/0 2024-09-16T10:58:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-c con 0x7f2c5000d260 addr v2:172.21.15.143:3300/0 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): picked mon.noname-b con 0x7f2c5000a570 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6409fb80 con 0x7f2c5000dd40 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c5000d260 2024-09-16T10:58:16.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409b970 con 0x7f2c5000a570 2024-09-16T10:58:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c617fa700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:16.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.551+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:16.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:16.755+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T10:58:17.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:17.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c5000d260 auth_method 0 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:58:17.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.159+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T10:58:17.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806092014 0 0) 0x7f2c6409fb80 con 0x7f2c5000dd40 2024-09-16T10:58:17.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 10 monclient(hunting): _negotiate 2024-09-16T10:58:17.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 10 monclient(hunting): my global_id is 64174 2024-09-16T10:58:17.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.227+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4c0035d0 con 0x7f2c5000dd40 2024-09-16T10:58:17.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3466292115 0 0) 0x7f2c4c0035d0 con 0x7f2c5000dd40 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c5000a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c5000a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 msgr2=0x7f2c5000d670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 --2- 172.21.15.5:0/944812328 >> v2:172.21.15.143:3300/0 conn(0x7f2c5000d260 0x7f2c5000d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 monclient: found mon.noname-c-legacy 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500136a0 con 0x7f2c5000dd40 2024-09-16T10:58:17.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f2c58004460 con 0x7f2c5000dd40 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c68885700 5 monclient: authenticate success, global_id 64174 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: dump: 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.244 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 >> v1:172.21.15.143:6789/0 conn(0x7f2c5000dd40 legacy=0x7f2c50013d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c4c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): picked mon.c con 0x7f2c5000a570 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:17.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c004050 0x7f2c4c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): picked mon.b con 0x7f2c4c004050 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4c004db0 0x7f2c4c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): picked mon.a con 0x7f2c4c004db0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409c030 con 0x7f2c4c004db0 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409b970 con 0x7f2c4c004050 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c00bb90 con 0x7f2c5000a570 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c627fc700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:58:17.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c68885700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4c004db0 msgr2=0x7f2c4c005230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c63fff700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4c004db0 0x7f2c4c005230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c4c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c5000a570 auth_method 0 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c004050 0x7f2c4c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c4c004050 auth_method 0 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient(hunting): handle_auth_done global_id 64174 payload 293 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c004050 msgr2=0x7f2c4c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c004050 0x7f2c4c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4c004db0 msgr2=0x7f2c4c005230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4c004db0 0x7f2c4c005230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 monclient: found mon.c 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 1 -- 172.21.15.5:0/944812328 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500136a0 con 0x7f2c5000a570 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.235+0000 7f2c637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c637fe700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c4c003a20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c4c0058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133b8bf587d56ad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5801d070 con 0x7f2c5000a570 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: dump: 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.250 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5800c040 con 0x7f2c5000a570 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 10 monclient: get_monmap_and_config success 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 4 set_mon_vals no callback set 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 1 -- 172.21.15.5:0/944812328 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58018df0 con 0x7f2c5000a570 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c627fc700 10 monclient: dump: 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.251 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.252 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 10 monclient: shutdown 2024-09-16T10:58:17.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 -- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 msgr2=0x7f2c4c003a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 --2- 172.21.15.5:0/944812328 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c5000a570 0x7f2c4c003a20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c58000c00 tx=0x7f2c4c0058a0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c63fff700 1 -- 172.21.15.5:0/944812328 reap_dead start 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 -- 172.21.15.5:0/944812328 shutdown_connections 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 -- 172.21.15.5:0/944812328 >> 172.21.15.5:0/944812328 conn(0x7f2c640a45d0 msgr2=0x7f2c640a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 -- 172.21.15.5:0/944812328 shutdown_connections 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.239+0000 7f2c68885700 1 -- 172.21.15.5:0/944812328 wait complete. 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: build_initial_monmap 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: monmap: 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:17.259 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 librados: starting msgr at 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 librados: starting objecter 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 Processor -- start 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- start start 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 librados: setting wanted keys 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 librados: calling monclient init 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: init 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-c con 0x7f2c641ce800 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c641397a0 0x7f2c64122d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-b con 0x7f2c641397a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 0x7f2c64123700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): picked mon.noname-a con 0x7f2c64123280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6409b0d0 con 0x7f2c64123280 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c641248c0 con 0x7f2c641397a0 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64139c20 con 0x7f2c641ce800 2024-09-16T10:58:17.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c63fff700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 msgr2=0x7f2c64123700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient(hunting): authenticate will time out at 1355.821938s 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c63fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 0x7f2c64123700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60868/0 (socket says 172.21.15.5:60868) 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 -- 172.21.15.5:0/1624945252 learned_addr learned my addr 172.21.15.5:0/1624945252 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): get_auth_request con 0x7f2c641ce800 auth_method 0 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c641397a0 0x7f2c64122d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request con 0x7f2c641397a0 auth_method 0 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient(hunting): handle_auth_done global_id 54311 payload 824 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 -- 172.21.15.5:0/1624945252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c641397a0 msgr2=0x7f2c64122d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c641397a0 0x7f2c64122d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 -- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 msgr2=0x7f2c64123700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 0x7f2c64123700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 monclient: found mon.noname-c 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 -- 172.21.15.5:0/1624945252 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c640a2990 con 0x7f2c641ce800 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c637fe700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c580091a0 tx=0x7f2c58004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 client.?.objecter ms_handle_connect 0x7f2c641ce800 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c58003a80 con 0x7f2c641ce800 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: dump: 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c58018de0 con 0x7f2c641ce800 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 5 monclient: authenticate success, global_id 54311 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c580226b0 con 0x7f2c641ce800 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _renew_subs 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c64126940 con 0x7f2c641ce800 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c61ffb700 4 set_mon_vals no callback set 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c4bfff700 10 monclient: dump: 2024-09-16T10:58:17.266 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 client.54311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _renew_subs 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c64139f40 con 0x7f2c641ce800 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.243+0000 7f2c68885700 1 librados: init done 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2c58022810 con 0x7f2c641ce800 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 1 --2- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 0x7f2c5003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(347..347 src has 57..347) v4 ==== 5317+0+0 (secure 0 0 0) 0x7f2c5804a730 con 0x7f2c641ce800 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 10 client.54311.objecter ms_dispatch 0x7f2c64000fd0 osd_map(347..347 src has 57..347) v4 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 3 client.54311.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 3 client.54311.objecter handle_osd_map decoding full epoch 347 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 20 client.54311.objecter dump_active .. 0 homeless 2024-09-16T10:58:17.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c68885700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c68885700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c68885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6409ad40 con 0x7f2c641ce800 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 0x7f2c5003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c62ffd700 10 monclient: get_auth_request con 0x7f2c50038860 auth_method 0 2024-09-16T10:58:17.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c4bfff700 10 client.54311.objecter ms_handle_connect 0x7f2c50038860 2024-09-16T10:58:17.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.247+0000 7f2c62ffd700 1 --2- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 0x7f2c5003ad20 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f2c641244c0 tx=0x7f2c54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.251+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2c58030080 con 0x7f2c641ce800 2024-09-16T10:58:17.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.251+0000 7f2c4bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.251+0000 7f2c4bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:58:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:58:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:58:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2c64125210 con 0x7f2c641ce800 2024-09-16T10:58:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c4bfff700 1 -- 172.21.15.5:0/1624945252 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f2c58043610 con 0x7f2c641ce800 2024-09-16T10:58:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c4bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:58:17.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c4bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:58:17.523 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:58:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 librados: watch_flush enter 2024-09-16T10:58:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 librados: watch_flush exit 2024-09-16T10:58:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 20 client.54311.objecter shutdown clearing up homeless session... 2024-09-16T10:58:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 client.54311.objecter successfully canceled tick 2024-09-16T10:58:17.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 msgr2=0x7f2c5003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 0x7f2c5003ad20 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f2c641244c0 tx=0x7f2c54008040 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 10 monclient: shutdown 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 msgr2=0x7f2c64139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c580091a0 tx=0x7f2c58004770 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 shutdown_connections 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c50038860 0x7f2c5003ad20 unknown :-1 s=CLOSED pgs=3813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c64123280 0x7f2c64123700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c641397a0 0x7f2c64122d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 --2- 172.21.15.5:0/1624945252 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c641ce800 0x7f2c64139260 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 >> 172.21.15.5:0/1624945252 conn(0x7f2c640a45d0 msgr2=0x7f2c6409cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 shutdown_connections 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.519+0000 7f2c68885700 1 -- 172.21.15.5:0/1624945252 wait complete. 2024-09-16T10:58:17.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.523+0000 7f2c68885700 1 librados: shutdown 2024-09-16T10:58:17.538 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T10:58:17.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T10:58:17.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:58:17.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: get_monmap_and_config 2024-09-16T10:58:17.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: build_initial_monmap 2024-09-16T10:58:17.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: monmap: 2024-09-16T10:58:17.648 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:17.648 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:58:17.647305+0000 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:58:17.647305+0000 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:58:17.649 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 1 Processor -- start 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 1 -- start start 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: init 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.643+0000 7f8194bb8700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81900a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 0x7f819013ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-a con 0x7f81900a3770 addr v2:172.21.15.5:3300/0 2024-09-16T10:58:17.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-b con 0x7f819013b1c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900a3db0 con 0x7f81900a2c80 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81900a4b70 con 0x7f81900a3770 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81900a0850 con 0x7f819013b1c0 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient(hunting): authenticate will time out at 1356.224116s 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818effd700 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 msgr2=0x7f819013ac80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 0x7f819013ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60872/0 (socket says 172.21.15.5:60872) 2024-09-16T10:58:17.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 -- 172.21.15.5:0/194257305 learned_addr learned my addr 172.21.15.5:0/194257305 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): get_auth_request con 0x7f819013b1c0 auth_method 0 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient(hunting): handle_auth_done global_id 54314 payload 274 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 -- 172.21.15.5:0/194257305 >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 msgr2=0x7f819013ac80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 --2- 172.21.15.5:0/194257305 >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 0x7f819013ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 -- 172.21.15.5:0/194257305 >> v1:172.21.15.143:6789/0 conn(0x7f81900a2c80 legacy=0x7f81900a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 monclient: found mon.noname-b 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 -- 172.21.15.5:0/194257305 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81900a4180 con 0x7f819013b1c0 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818ffff700 1 --2- 172.21.15.5:0/194257305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8188009910 tx=0x7f8188009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c9caee8f2e1309 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 1 -- 172.21.15.5:0/194257305 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818800ba60 con 0x7f819013b1c0 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: dump: 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.665 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 5 monclient: authenticate success, global_id 54314 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 1 -- 172.21.15.5:0/194257305 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818800bbc0 con 0x7f819013b1c0 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient: get_monmap_and_config success 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 4 set_mon_vals no callback set 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 1 -- 172.21.15.5:0/194257305 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188009e60 con 0x7f819013b1c0 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f818e7fc700 10 monclient: dump: 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.667 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient: shutdown 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- 172.21.15.5:0/194257305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 msgr2=0x7f819013d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 --2- 172.21.15.5:0/194257305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8188009910 tx=0x7f8188009c20 comp rx=0 tx=0).stop 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- 172.21.15.5:0/194257305 shutdown_connections 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 --2- 172.21.15.5:0/194257305 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f819013b1c0 0x7f819013d5b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 --2- 172.21.15.5:0/194257305 >> v2:172.21.15.5:3300/0 conn(0x7f81900a3770 0x7f819013ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- 172.21.15.5:0/194257305 >> 172.21.15.5:0/194257305 conn(0x7f81900a4d80 msgr2=0x7f81900a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- 172.21.15.5:0/194257305 shutdown_connections 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 -- 172.21.15.5:0/194257305 wait complete. 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient: build_initial_monmap 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 10 monclient: monmap: 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:58:17.669 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 librados: starting msgr at 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.647+0000 7f8194bb8700 1 librados: starting objecter 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 Processor -- start 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- start start 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 librados: setting wanted keys 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 librados: calling monclient init 2024-09-16T10:58:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: init 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 0x7f819006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-c con 0x7f81900a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81900a3770 0x7f8190134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-a con 0x7f81900a3770 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f819013b1c0 0x7f81901350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): picked mon.noname-b con 0x7f819013b1c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819011f9d0 con 0x7f81900a3770 2024-09-16T10:58:17.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819009f590 con 0x7f819013b1c0 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81901200a0 con 0x7f81900a2c80 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient(hunting): authenticate will time out at 1356.229085s 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818effd700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81900a3770 msgr2=0x7f8190134bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818effd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81900a3770 0x7f8190134bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 0x7f819006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 0x7f819006f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60876/0 (socket says 172.21.15.5:60876) 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 -- 172.21.15.5:0/3673044556 learned_addr learned my addr 172.21.15.5:0/3673044556 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): get_auth_request con 0x7f81900a2c80 auth_method 0 2024-09-16T10:58:17.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f819013b1c0 0x7f81901350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): get_auth_request con 0x7f819013b1c0 auth_method 0 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient(hunting): handle_auth_done global_id 54317 payload 824 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f819013b1c0 msgr2=0x7f81901350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f819013b1c0 0x7f81901350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81900a3770 msgr2=0x7f8190134bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81900a3770 0x7f8190134bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 monclient: found mon.noname-c 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8188009570 con 0x7f81900a2c80 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 client.?.objecter ms_handle_connect 0x7f81900a2c80 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818f7fe700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 0x7f819006f7e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f817800c500 tx=0x7f817800c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81780052d0 con 0x7f81900a2c80 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: dump: 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.675 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 5 monclient: authenticate success, global_id 54317 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8178005430 con 0x7f81900a2c80 2024-09-16T10:58:17.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _renew_subs 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8190120cc0 con 0x7f81900a2c80 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81780055a0 con 0x7f81900a2c80 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8187fff700 10 monclient: dump: 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.678 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 client.54317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:58:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _renew_subs 2024-09-16T10:58:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8190121050 con 0x7f81900a2c80 2024-09-16T10:58:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f818dffb700 4 set_mon_vals no callback set 2024-09-16T10:58:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 1 librados: init done 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.651+0000 7f8194bb8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8194bb8700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 0x7f81901ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8190067380 con 0x7f81901ca4c0 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 0x7f81901ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient: get_auth_request con 0x7f81901ca4c0 auth_method 0 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 10 monclient(hunting): handle_auth_done global_id 54320 payload 824 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f81780038e0 con 0x7f81900a2c80 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f819009fa10 con 0x7f81901ca4c0 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 --2- 172.21.15.5:0/3673044556 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8170036f10 0x7f81700393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818effd700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 0x7f81901ca8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f819006ff00 tx=0x7f8180009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 client.54317.objecter ms_handle_connect 0x7f81901ca4c0 2024-09-16T10:58:17.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 client.54317.objecter resend_mon_ops 2024-09-16T10:58:17.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(348..348 src has 57..348) v4 ==== 4932+0+0 (secure 0 0 0) 0x7f8178047070 con 0x7f81900a2c80 2024-09-16T10:58:17.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 client.54317.objecter ms_dispatch 0x7f8190000fd0 osd_map(348..348 src has 57..348) v4 2024-09-16T10:58:17.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 3 client.54317.objecter handle_osd_map got epochs [348,348] > 0 2024-09-16T10:58:17.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 3 client.54317.objecter handle_osd_map decoding full epoch 348 2024-09-16T10:58:17.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 20 client.54317.objecter dump_active .. 0 homeless 2024-09-16T10:58:17.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000cdb0 con 0x7f81901ca4c0 2024-09-16T10:58:17.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 monclient: dump: 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.686 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.687 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.687 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.687 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.687 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818ffff700 1 --2- 172.21.15.5:0/3673044556 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8170036f10 0x7f81700393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818ffff700 10 monclient: get_auth_request con 0x7f8170036f10 auth_method 0 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f8180015470 con 0x7f81901ca4c0 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 msgr2=0x7f81901ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 0x7f81901ca8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f819006ff00 tx=0x7f8180009ed0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 msgr2=0x7f81901ca8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:58:17.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901ca4c0 0x7f81901ca8a0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f819006ff00 tx=0x7f8180009ed0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f818ffff700 1 --2- 172.21.15.5:0/3673044556 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8170036f10 0x7f81700393d0 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f8188009c60 tx=0x7f8188008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.655+0000 7f8187fff700 10 client.54317.objecter ms_handle_connect 0x7f8170036f10 2024-09-16T10:58:17.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:58:17.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 0x7f8190139f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819006fd80 con 0x7f81901cade0 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 0x7f8190139f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient: get_auth_request con 0x7f81901cade0 auth_method 0 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 10 monclient(hunting): handle_auth_done global_id 54323 payload 824 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 1 -- 172.21.15.5:0/3673044556 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f81900a4400 con 0x7f81901cade0 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 client.54317.objecter ms_handle_connect 0x7f81901cade0 2024-09-16T10:58:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 client.54317.objecter resend_mon_ops 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818effd700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 0x7f8190139f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8180014000 tx=0x7f8180005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180023410 con 0x7f81901cade0 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 monclient: dump: 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.696 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7f8180023570 con 0x7f81901cade0 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 msgr2=0x7f8190139f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 0x7f8190139f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8180014000 tx=0x7f8180005e50 comp rx=0 tx=0).stop 2024-09-16T10:58:17.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 msgr2=0x7f8190139f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8187fff700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81901cade0 0x7f8190139f90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 42, 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:58:17.698 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.699 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.700 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:58:17.701 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:58:17.702 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.703 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.704 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.705 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:58:17.707 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:58:17.708 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:58:17.709 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:58:17.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.711 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.712 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:58:17.713 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:58:17.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 librados: watch_flush enter 2024-09-16T10:58:17.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 librados: watch_flush exit 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 20 client.54317.objecter shutdown clearing up homeless session... 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 client.54317.objecter successfully canceled tick 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8170036f10 msgr2=0x7f81700393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 --2- 172.21.15.5:0/3673044556 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8170036f10 0x7f81700393d0 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f8188009c60 tx=0x7f8188008030 comp rx=0 tx=0).stop 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 10 monclient: shutdown 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 msgr2=0x7f819006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 --2- 172.21.15.5:0/3673044556 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f81900a2c80 0x7f819006f7e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f817800c500 tx=0x7f817800c810 comp rx=0 tx=0).stop 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818ffff700 1 -- 172.21.15.5:0/3673044556 reap_dead start 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f818ffff700 1 -- 172.21.15.5:0/3673044556 reap_dead start 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 shutdown_connections 2024-09-16T10:58:17.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 >> 172.21.15.5:0/3673044556 conn(0x7f81900a4d80 msgr2=0x7f81900a6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 shutdown_connections 2024-09-16T10:58:17.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 -- 172.21.15.5:0/3673044556 wait complete. 2024-09-16T10:58:17.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.667+0000 7f8194bb8700 1 librados: shutdown 2024-09-16T10:58:17.716 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:58:17.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T10:58:17.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:58:17.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: get_monmap_and_config 2024-09-16T10:58:17.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: build_initial_monmap 2024-09-16T10:58:17.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: monmap: 2024-09-16T10:58:17.838 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:17.838 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:58:17.837470+0000 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:58:17.837470+0000 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:58:17.839 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 Processor -- start 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 -- start start 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: init 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9300a2c80 0x7fd9300a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fd9300a2c80 addr v2:172.21.15.143:3300/0 2024-09-16T10:58:17.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 0x7fd930135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fd9300a35a0 addr v2:172.21.15.5:3300/0 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd930135f80 addr v1:172.21.15.5:6789/0 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9300a3db0 con 0x7fd930135f80 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9300a4b70 con 0x7fd9300a35a0 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9300a0850 con 0x7fd9300a2c80 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92ffff700 10 monclient(hunting): authenticate will time out at 1356.414499s 2024-09-16T10:58:17.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92e7fc700 1 -- >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 msgr2=0x7fd930135a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92e7fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 0x7fd930135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92f7fe700 1 -- >> v1:172.21.15.5:6789/0 conn(0x7fd930135f80 legacy=0x7fd93013cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9300a2c80 0x7fd9300a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9300a2c80 0x7fd9300a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:57898/0 (socket says 172.21.15.5:57898) 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 -- 172.21.15.5:0/1151578578 learned_addr learned my addr 172.21.15.5:0/1151578578 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): get_auth_request con 0x7fd9300a2c80 auth_method 0 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient(hunting): handle_auth_done global_id 64186 payload 274 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 -- 172.21.15.5:0/1151578578 >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 msgr2=0x7fd930135a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 --2- 172.21.15.5:0/1151578578 >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 0x7fd930135a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 -- 172.21.15.5:0/1151578578 >> v1:172.21.15.5:6789/0 conn(0x7fd930135f80 legacy=0x7fd93013cc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 monclient: found mon.noname-c 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:58:17.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 -- 172.21.15.5:0/1151578578 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9300a4180 con 0x7fd9300a2c80 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.835+0000 7fd92effd700 1 --2- 172.21.15.5:0/1151578578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9300a2c80 0x7fd9300a3060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd9300a81c0 tx=0x7fd920009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=922e29476079e6a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 1 -- 172.21.15.5:0/1151578578 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920004230 con 0x7fd9300a2c80 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: dump: 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.848 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 1 -- 172.21.15.5:0/1151578578 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920004390 con 0x7fd9300a2c80 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 5 monclient: authenticate success, global_id 64186 2024-09-16T10:58:17.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 10 monclient: get_monmap_and_config success 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 1 -- 172.21.15.5:0/1151578578 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200036c0 con 0x7fd9300a2c80 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 4 set_mon_vals no callback set 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92dffb700 10 monclient: dump: 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.850 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 10 monclient: shutdown 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- 172.21.15.5:0/1151578578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9300a2c80 msgr2=0x7fd9300a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 --2- 172.21.15.5:0/1151578578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9300a2c80 0x7fd9300a3060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd9300a81c0 tx=0x7fd920009bf0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- 172.21.15.5:0/1151578578 shutdown_connections 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 --2- 172.21.15.5:0/1151578578 >> v2:172.21.15.5:3300/0 conn(0x7fd9300a35a0 0x7fd930135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 --2- 172.21.15.5:0/1151578578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9300a2c80 0x7fd9300a3060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- 172.21.15.5:0/1151578578 >> 172.21.15.5:0/1151578578 conn(0x7fd9300a4d80 msgr2=0x7fd9300a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- 172.21.15.5:0/1151578578 shutdown_connections 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- 172.21.15.5:0/1151578578 wait complete. 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 10 monclient: build_initial_monmap 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 10 monclient: monmap: 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:58:17.852 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:58:17.853 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 librados: starting msgr at 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 librados: starting objecter 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 Processor -- start 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 -- start start 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 librados: setting wanted keys 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 1 librados: calling monclient init 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.839+0000 7fd92ffff700 10 monclient: init 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _reopen_session rank -1 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:58:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 0x7fd9301ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fd9300a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9300a35a0 0x7fd930134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fd9300a35a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930135f80 0x7fd930135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fd930135f80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9301266f0 con 0x7fd9300a35a0 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9301cade0 con 0x7fd930135f80 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9301cb620 con 0x7fd9300a2c80 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): _renew_subs 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): authenticate will time out at 1356.420607s 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 1 -- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9300a35a0 msgr2=0x7fd930134fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3300/0 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9300a35a0 0x7fd930134fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 0x7fd9301ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 0x7fd9301ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60890/0 (socket says 172.21.15.5:60890) 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 -- 172.21.15.5:0/4142402055 learned_addr learned my addr 172.21.15.5:0/4142402055 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): get_auth_request con 0x7fd9300a2c80 auth_method 0 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930135f80 0x7fd930135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 10 monclient(hunting): get_auth_request con 0x7fd930135f80 auth_method 0 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient(hunting): handle_auth_done global_id 54326 payload 824 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930135f80 msgr2=0x7fd930135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930135f80 0x7fd930135510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9300a35a0 msgr2=0x7fd930134fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9300a35a0 0x7fd930134fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient: _finish_hunting 0 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 monclient: found mon.noname-c 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920009570 con 0x7fd9300a2c80 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92f7fe700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930135f80 0x7fd930135510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 client.?.objecter ms_handle_connect 0x7fd9300a2c80 2024-09-16T10:58:17.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92effd700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 0x7fd9301ca8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd920009c30 tx=0x7fd920004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 client.?.objecter resend_mon_ops 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000b860 con 0x7fd9300a2c80 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: dump: 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.865 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: _finish_auth 0 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: _check_auth_tickets 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 5 monclient: authenticate success, global_id 54326 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92001f070 con 0x7fd9300a2c80 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:58:17.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000bbc0 con 0x7fd9300a2c80 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 10 monclient: dump: 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.867 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _renew_subs 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd930126410 con 0x7fd9300a2c80 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92d7fa700 4 set_mon_vals no callback set 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 client.54326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _renew_subs 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:58:17.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd930126130 con 0x7fd9300a2c80 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 librados: init done 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 0x7fd930122db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd930067380 con 0x7fd9301229d0 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd920021030 con 0x7fd9300a2c80 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 0x7fd930122db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 10 monclient: get_auth_request con 0x7fd9301229d0 auth_method 0 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.843+0000 7fd92e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 --2- 172.21.15.5:0/4142402055 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd91c036f10 0x7fd91c0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(348..348 src has 57..348) v4 ==== 4932+0+0 (secure 0 0 0) 0x7fd920049340 con 0x7fd9300a2c80 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 client.54326.objecter ms_dispatch 0x7fd930000fd0 osd_map(348..348 src has 57..348) v4 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 3 client.54326.objecter handle_osd_map got epochs [348,348] > 0 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 3 client.54326.objecter handle_osd_map decoding full epoch 348 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_done global_id 64192 payload 824 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9300a39b0 con 0x7fd9301229d0 2024-09-16T10:58:17.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 20 client.54326.objecter dump_active .. 0 homeless 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92f7fe700 1 --2- 172.21.15.5:0/4142402055 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd91c036f10 0x7fd91c0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92f7fe700 10 monclient: get_auth_request con 0x7fd91c036f10 auth_method 0 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 client.54326.objecter ms_handle_connect 0x7fd9301229d0 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 client.54326.objecter resend_mon_ops 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92e7fc700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 0x7fd930122db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd924003a30 tx=0x7fd924009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd924003ab0 con 0x7fd9301229d0 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 monclient: dump: 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 client.54326.objecter ms_handle_connect 0x7fd91c036f10 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd92f7fe700 1 --2- 172.21.15.5:0/4142402055 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd91c036f10 0x7fd91c0393d0 secure :-1 s=READY pgs=3815 cs=0 l=1 rev1=1 crypto rx=0x7fd930124d50 tx=0x7fd9280094d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd924003c10 con 0x7fd9301229d0 2024-09-16T10:58:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:58:17.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:58:17.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 msgr2=0x7fd930122db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 0x7fd930122db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd924003a30 tx=0x7fd924009f30 comp rx=0 tx=0).stop 2024-09-16T10:58:17.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 msgr2=0x7fd930122db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:58:17.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.847+0000 7fd917fff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301229d0 0x7fd930122db0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd924003a30 tx=0x7fd924009f30 comp rx=0 tx=0).stop 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92ffff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 0x7fd93011e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93009f570 con 0x7fd9301232f0 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 0x7fd93011e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient: get_auth_request con 0x7fd9301232f0 auth_method 0 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:58:17.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 10 monclient(hunting): handle_auth_done global_id 64195 payload 824 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 1 -- 172.21.15.5:0/4142402055 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9300a4400 con 0x7fd9301232f0 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd92e7fc700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 0x7fd93011e8d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd924009fd0 tx=0x7fd924005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 10 client.54326.objecter ms_handle_connect 0x7fd9301232f0 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 10 client.54326.objecter resend_mon_ops 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92401a990 con 0x7fd9301232f0 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:58:17.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.859+0000 7fd917fff700 10 monclient: dump: 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:58:17.877 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7fd924005940 con 0x7fd9301232f0 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 msgr2=0x7fd93011e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 0x7fd93011e8d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd924009fd0 tx=0x7fd924005ad0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 msgr2=0x7fd93011e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd917fff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9301232f0 0x7fd93011e8d0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd924009fd0 tx=0x7fd924005ad0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.878 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 42, 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:58:17.879 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.880 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.881 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:58:17.882 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:58:17.883 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:58:17.884 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.885 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.887 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.888 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.890 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:58:17.891 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:58:17.892 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.893 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:58:17.894 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:58:17.895 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:58:17.895 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:58:17.895 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:58:17.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 10 librados: watch_flush enter 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 10 librados: watch_flush exit 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 20 client.54326.objecter shutdown clearing up homeless session... 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 10 client.54326.objecter successfully canceled tick 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd91c036f10 msgr2=0x7fd91c0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 --2- 172.21.15.5:0/4142402055 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd91c036f10 0x7fd91c0393d0 secure :-1 s=READY pgs=3815 cs=0 l=1 rev1=1 crypto rx=0x7fd930124d50 tx=0x7fd9280094d0 comp rx=0 tx=0).stop 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 10 monclient: shutdown 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 msgr2=0x7fd9301ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 --2- 172.21.15.5:0/4142402055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd9300a2c80 0x7fd9301ca8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd920009c30 tx=0x7fd920004770 comp rx=0 tx=0).stop 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92f7fe700 1 -- 172.21.15.5:0/4142402055 reap_dead start 2024-09-16T10:58:17.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92f7fe700 1 -- 172.21.15.5:0/4142402055 reap_dead start 2024-09-16T10:58:17.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 shutdown_connections 2024-09-16T10:58:17.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 >> 172.21.15.5:0/4142402055 conn(0x7fd9300a4d80 msgr2=0x7fd9300a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:58:17.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 shutdown_connections 2024-09-16T10:58:17.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 -- 172.21.15.5:0/4142402055 wait complete. 2024-09-16T10:58:17.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:58:17.863+0000 7fd92ffff700 1 librados: shutdown 2024-09-16T10:58:17.899 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T10:58:17.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T10:59:00.765 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T10:59:00.760+0000 7f35b1aee700 -1 snap_mapper.add_oid found existing snaps mapped on 60:e451c8d2:test-rados-api-smithi143-24525-76::foo:28, removing 2024-09-16T10:59:00.766 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:59:00.760+0000 7f482b588700 -1 snap_mapper.add_oid found existing snaps mapped on 60:e451c8d2:test-rados-api-smithi143-24525-76::foo:28, removing 2024-09-16T10:59:04.623 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T10:59:04.619+0000 7f483559c700 -1 snap_mapper.add_oid found existing snaps mapped on 60:a4f0f40a:test-rados-api-smithi143-24525-78::foo:29, removing 2024-09-16T10:59:04.624 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-16T10:59:04.618+0000 7feda7445700 -1 snap_mapper.add_oid found existing snaps mapped on 60:a4f0f40a:test-rados-api-smithi143-24525-78::foo:29, removing 2024-09-16T10:59:29.724 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-09-16T10:59:29.717+0000 7f1f7cf8e700 -1 snap_mapper.add_oid found existing snaps mapped on 220:d04ac100:test-rados-api-smithi143-24525-80::foo:2, removing 2024-09-16T10:59:29.724 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T10:59:29.719+0000 7f0fa09bc700 -1 snap_mapper.add_oid found existing snaps mapped on 220:d04ac100:test-rados-api-smithi143-24525-80::foo:2, removing 2024-09-16T10:59:47.900 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T10:59:47.901 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T10:59:47.901 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T10:59:47.901 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T10:59:47.904 INFO:tasks.ceph.mon.a:Started 2024-09-16T10:59:47.905 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:59:47.905 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:59:48.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: get_monmap_and_config 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: build_initial_monmap 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: monmap: 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:48.013091+0000 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:48.013091+0000 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:48.014 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:48.015 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:48.015 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:48.015 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:48.015 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:48.015 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 Processor -- start 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 -- start start 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: init 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-a con 0x7f12380a2eb0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-b con 0x7f12380a4280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12380a3d10 0x7f1238135c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-c con 0x7f12380a3d10 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:48.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380a1e50 con 0x7f12380a2eb0 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123809fdc0 con 0x7f12380a3d10 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123809fae0 con 0x7f12380a4280 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient(hunting): authenticate will time out at 1446.590021s 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:42472/0 (socket says 172.21.15.5:42472) 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 -- 172.21.15.5:0/3873410418 learned_addr learned my addr 172.21.15.5:0/3873410418 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): get_auth_request con 0x7f12380a4280 auth_method 0 2024-09-16T10:59:48.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 10 monclient(hunting): get_auth_request con 0x7f12380a2eb0 auth_method 0 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.143:3300/0 conn(0x7f12380a3d10 0x7f1238135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): get_auth_request con 0x7f12380a3d10 auth_method 0 2024-09-16T10:59:48.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 1 -- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 msgr2=0x7f12380a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 1 -- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 msgr2=0x7f12380a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1237fff700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient(hunting): handle_auth_done global_id 54329 payload 274 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 -- 172.21.15.5:0/3873410418 >> v2:172.21.15.143:3300/0 conn(0x7f12380a3d10 msgr2=0x7f1238135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.143:3300/0 conn(0x7f12380a3d10 0x7f1238135c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 -- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 msgr2=0x7f12380a3290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 monclient: found mon.noname-b 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 -- 172.21.15.5:0/3873410418 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123809efb0 con 0x7f12380a4280 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f12377fe700 1 --2- 172.21.15.5:0/3873410418 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1238071b10 tx=0x7f122c009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed7de36811dd932 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 1 -- 172.21.15.5:0/3873410418 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c011830 con 0x7f12380a4280 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: dump: 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:48.030 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: _finish_auth 0 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 1 -- 172.21.15.5:0/3873410418 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c011990 con 0x7f12380a4280 2024-09-16T10:59:48.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 5 monclient: authenticate success, global_id 54329 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 1 -- 172.21.15.5:0/3873410418 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c011d20 con 0x7f12380a4280 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f123ea42700 4 set_mon_vals no callback set 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.009+0000 7f1236ffd700 10 monclient: dump: 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: shutdown 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- 172.21.15.5:0/3873410418 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 msgr2=0x7f12380a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- 172.21.15.5:0/3873410418 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1238071b10 tx=0x7f122c009bf0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- 172.21.15.5:0/3873410418 shutdown_connections 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.143:3300/0 conn(0x7f12380a3d10 0x7f1238135c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- 172.21.15.5:0/3873410418 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a4280 0x7f12380a37d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- 172.21.15.5:0/3873410418 >> v2:172.21.15.5:3300/0 conn(0x7f12380a2eb0 0x7f12380a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- 172.21.15.5:0/3873410418 >> 172.21.15.5:0/3873410418 conn(0x7f12380a4fb0 msgr2=0x7f12380a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- 172.21.15.5:0/3873410418 shutdown_connections 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- 172.21.15.5:0/3873410418 wait complete. 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: build_initial_monmap 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: monmap: 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:48.034 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 librados: starting msgr at 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 librados: starting objecter 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 Processor -- start 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- start start 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 librados: setting wanted keys 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 librados: calling monclient init 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: init 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a2eb0 0x7f12381cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-c con 0x7f12380a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-a con 0x7f12380a3d10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 0x7f12381222a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): picked mon.noname-b con 0x7f12380a4280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1238134f70 con 0x7f12380a3d10 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380a6d40 con 0x7f12380a4280 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1238125f20 con 0x7f12380a2eb0 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f123ea42700 10 monclient(hunting): authenticate will time out at 1446.595809s 2024-09-16T10:59:48.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.013+0000 7f12377fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57236/0 (socket says 172.21.15.5:57236) 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 -- 172.21.15.5:0/3501056915 learned_addr learned my addr 172.21.15.5:0/3501056915 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 10 monclient(hunting): get_auth_request con 0x7f12380a3d10 auth_method 0 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a2eb0 0x7f12381cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 0x7f12381222a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 -- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 msgr2=0x7f12381cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 -- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 msgr2=0x7f12381cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:59:48.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): get_auth_request con 0x7f12380a4280 auth_method 0 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f12377fe700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): get_auth_request con 0x7f12380a2eb0 auth_method 0 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:48.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient(hunting): handle_auth_done global_id 64201 payload 824 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 -- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a2eb0 msgr2=0x7f12381cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a2eb0 0x7f12381cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 -- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 msgr2=0x7f12381cb390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient: _finish_hunting 0 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 monclient: found mon.noname-b 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 -- 172.21.15.5:0/3501056915 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c009570 con 0x7f12380a4280 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123cfdf700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 0x7f12381222a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f123000efc0 tx=0x7f1230009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 client.?.objecter ms_handle_connect 0x7f12380a4280 2024-09-16T10:59:48.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123000cb10 con 0x7f12380a4280 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: dump: 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:48.040 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: _finish_auth 0 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 5 monclient: authenticate success, global_id 64201 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123000f040 con 0x7f12380a4280 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _renew_subs 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12381243b0 con 0x7f12380a4280 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230014480 con 0x7f12380a4280 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:48.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 monclient: dump: 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1235ffb700 4 set_mon_vals no callback set 2024-09-16T10:59:48.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 client.64201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _renew_subs 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1238123ea0 con 0x7f12380a4280 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 1 librados: init done 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f12300145e0 con 0x7f12380a4280 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 --2- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 0x7f1224039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(391..391 src has 344..391) v4 ==== 5384+0+0 (secure 0 0 0) 0x7f12300488b0 con 0x7f12380a4280 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 10 client.64201.objecter ms_dispatch 0x7f1238000fd0 osd_map(391..391 src has 344..391) v4 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 3 client.64201.objecter handle_osd_map got epochs [391,391] > 0 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 3 client.64201.objecter handle_osd_map decoding full epoch 391 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1234ff9700 20 client.64201.objecter dump_active .. 0 homeless 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:48.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:59:48.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1238136130 con 0x7f12380a4280 2024-09-16T10:59:48.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 1 --2- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 0x7f1224039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:48.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.017+0000 7f1237fff700 10 monclient: get_auth_request con 0x7f1224036f60 auth_method 0 2024-09-16T10:59:48.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.021+0000 7f1234ff9700 10 client.64201.objecter ms_handle_connect 0x7f1224036f60 2024-09-16T10:59:48.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.021+0000 7f1237fff700 1 --2- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 0x7f1224039420 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7f1228009100 tx=0x7f1228008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:48.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.029+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1230018020 con 0x7f12380a4280 2024-09-16T10:59:48.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.033+0000 7f1234ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:48.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.033+0000 7f1234ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:48.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f123ea42700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:59:48.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f123ea42700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:48.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f123ea42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T10:59:48.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1238135c50 con 0x7f12380a4280 2024-09-16T10:59:48.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f1234ff9700 1 -- 172.21.15.5:0/3501056915 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f1230007510 con 0x7f12380a4280 2024-09-16T10:59:48.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f1234ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:48.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f1234ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:48.271 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:59:48.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.265+0000 7f123ea42700 10 librados: watch_flush enter 2024-09-16T10:59:48.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 10 librados: watch_flush exit 2024-09-16T10:59:48.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 20 client.64201.objecter shutdown clearing up homeless session... 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 10 client.64201.objecter successfully canceled tick 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 msgr2=0x7f1224039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 0x7f1224039420 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7f1228009100 tx=0x7f1228008040 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 10 monclient: shutdown 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 msgr2=0x7f12381222a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 0x7f12381222a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f123000efc0 tx=0x7f1230009f90 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 shutdown_connections 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12380a4280 0x7f12381222a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1224036f60 0x7f1224039420 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f12380a3d10 0x7f12381cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 --2- 172.21.15.5:0/3501056915 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f12380a2eb0 0x7f12381cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 >> 172.21.15.5:0/3501056915 conn(0x7f12380a4fb0 msgr2=0x7f123809de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:48.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 shutdown_connections 2024-09-16T10:59:48.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 -- 172.21.15.5:0/3501056915 wait complete. 2024-09-16T10:59:48.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:48.269+0000 7f123ea42700 1 librados: shutdown 2024-09-16T10:59:51.286 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:59:51.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.397+0000 7f0f05749700 10 monclient: get_monmap_and_config 2024-09-16T10:59:51.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.397+0000 7f0f05749700 10 monclient: build_initial_monmap 2024-09-16T10:59:51.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.397+0000 7f0f05749700 10 monclient: monmap: 2024-09-16T10:59:51.405 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:51.405 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:51.404284+0000 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:51.404284+0000 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 Processor -- start 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- start start 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient: init 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f0009b3e0 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f0009bed0 0x7f0f001378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-c con 0x7f0f0009bed0 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-a con 0x7f0f0009c2e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0009e350 con 0x7f0f0009b3e0 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0009ca20 con 0x7f0f0009c2e0 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f0009cd00 con 0x7f0f0009bed0 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient(hunting): authenticate will time out at 1449.981023s 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0efffff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0f0009b3e0 0x7f0f0009b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57794/0 (socket says 172.21.15.5:57794) 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0efffff700 1 -- 172.21.15.5:0/4127733171 learned_addr learned my addr 172.21.15.5:0/4127733171 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.143:3300/0 conn(0x7f0f0009bed0 0x7f0f001378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 10 monclient(hunting): get_auth_request con 0x7f0f0009c2e0 auth_method 0 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): get_auth_request con 0x7f0f0009bed0 auth_method 0 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 1 -- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 msgr2=0x7f0f00139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 1 -- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 msgr2=0x7f0f00139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:59:51.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f04f48700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_done global_id 64204 payload 274 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 -- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 msgr2=0x7f0f00139e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 -- 172.21.15.5:0/4127733171 >> v1:172.21.15.5:6789/0 conn(0x7f0f0009b3e0 legacy=0x7f0f0009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 monclient: found mon.noname-c 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 -- 172.21.15.5:0/4127733171 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0009cfe0 con 0x7f0f0009bed0 2024-09-16T10:59:51.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0ef77fe700 1 --2- 172.21.15.5:0/4127733171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f001378f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee8009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d1c26a0f1f1bc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 1 -- 172.21.15.5:0/4127733171 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8011830 con 0x7f0f0009bed0 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: dump: 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 1 -- 172.21.15.5:0/4127733171 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee8011990 con 0x7f0f0009bed0 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 5 monclient: authenticate success, global_id 64204 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 1 -- 172.21.15.5:0/4127733171 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8011d20 con 0x7f0f0009bed0 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 4 set_mon_vals no callback set 2024-09-16T10:59:51.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0eff7fe700 10 monclient: dump: 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:51.417 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 10 monclient: shutdown 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- 172.21.15.5:0/4127733171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 msgr2=0x7f0f001378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 --2- 172.21.15.5:0/4127733171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f001378f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee8009b60 comp rx=0 tx=0).stop 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- 172.21.15.5:0/4127733171 shutdown_connections 2024-09-16T10:59:51.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 --2- 172.21.15.5:0/4127733171 >> v2:172.21.15.5:3300/0 conn(0x7f0f0009c2e0 0x7f0f00139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 --2- 172.21.15.5:0/4127733171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f001378f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.401+0000 7f0f05749700 1 -- 172.21.15.5:0/4127733171 >> 172.21.15.5:0/4127733171 conn(0x7f0f000a45d0 msgr2=0x7f0f000a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 -- 172.21.15.5:0/4127733171 shutdown_connections 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 -- 172.21.15.5:0/4127733171 wait complete. 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient: build_initial_monmap 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient: monmap: 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:51.419 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:51.420 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:51.420 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:51.420 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 librados: starting msgr at 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 librados: starting objecter 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 Processor -- start 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 -- start start 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 librados: setting wanted keys 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 librados: calling monclient init 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient: init 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-a con 0x7f0f0009b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f00139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-b con 0x7f0f0009bed0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 0x7f0f0011e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient(hunting): picked mon.noname-c con 0x7f0f0009c2e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.405+0000 7f0f05749700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000a2430 con 0x7f0f0009b3e0 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57244/0 (socket says 172.21.15.5:57244) 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 -- 172.21.15.5:0/1162470097 learned_addr learned my addr 172.21.15.5:0/1162470097 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 0x7f0f0011e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00070f20 con 0x7f0f0009bed0 2024-09-16T10:59:51.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0009b0d0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient(hunting): authenticate will time out at 1449.988345s 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 10 monclient(hunting): get_auth_request con 0x7f0f0009b3e0 auth_method 0 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): get_auth_request con 0x7f0f0009c2e0 auth_method 0 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f00139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): get_auth_request con 0x7f0f0009bed0 auth_method 0 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 -- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 msgr2=0x7f0f001c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 -- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 msgr2=0x7f0f001c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:59:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient(hunting): handle_auth_done global_id 54335 payload 824 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 -- 172.21.15.5:0/1162470097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 msgr2=0x7f0f00139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f00139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 -- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 msgr2=0x7f0f001c8590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient: _finish_hunting 0 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 monclient: found mon.noname-c 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80094b0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0ef77fe700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f00139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f04f48700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 0x7f0f0011e000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8007cf0 tx=0x7f0ef800e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:51.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 client.?.objecter ms_handle_connect 0x7f0f0009c2e0 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8018070 con 0x7f0f0009c2e0 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: dump: 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:51.430 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: _finish_auth 0 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: _check_auth_tickets 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 5 monclient: authenticate success, global_id 54335 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _renew_subs 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8003a40 con 0x7f0f0009c2e0 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f00121fb0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8003e30 con 0x7f0f0009c2e0 2024-09-16T10:59:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efeffd700 4 set_mon_vals no callback set 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 monclient: dump: 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:51.432 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 client.54335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _renew_subs 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f001c9420 con 0x7f0f0009c2e0 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 librados: init done 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0ef800ea20 con 0x7f0f0009c2e0 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 --2- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 0x7f0ee0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(394..394 src has 344..394) v4 ==== 5304+0+0 (secure 0 0 0) 0x7f0ef8047de0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 10 client.54335.objecter ms_dispatch 0x7f0f00000fd0 osd_map(394..394 src has 344..394) v4 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 3 client.54335.objecter handle_osd_map got epochs [394,394] > 0 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 3 client.54335.objecter handle_osd_map decoding full epoch 394 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:51.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efd7fa700 20 client.54335.objecter dump_active .. 0 homeless 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0011fcb0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 1 --2- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 0x7f0ee0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.409+0000 7f0efffff700 10 monclient: get_auth_request con 0x7f0ee0036f60 auth_method 0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.413+0000 7f0efd7fa700 10 client.54335.objecter ms_handle_connect 0x7f0ee0036f60 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.413+0000 7f0efffff700 1 --2- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 0x7f0ee0039420 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0007fd0 tx=0x7f0ef0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.417+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0ef8014420 con 0x7f0f0009c2e0 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.417+0000 7f0efd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:51.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.417+0000 7f0efd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:51.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0f05749700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:59:51.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0f05749700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:51.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0f05749700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:51.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0f0009b7c0 con 0x7f0f0009c2e0 2024-09-16T10:59:51.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0efd7fa700 1 -- 172.21.15.5:0/1162470097 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f0ef803c080 con 0x7f0f0009c2e0 2024-09-16T10:59:51.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0efd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:51.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.653+0000 7f0efd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:51.660 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":42,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:59:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 10 librados: watch_flush enter 2024-09-16T10:59:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 10 librados: watch_flush exit 2024-09-16T10:59:51.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 20 client.54335.objecter shutdown clearing up homeless session... 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 10 client.54335.objecter successfully canceled tick 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 msgr2=0x7f0ee0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 0x7f0ee0039420 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0007fd0 tx=0x7f0ef0009040 comp rx=0 tx=0).stop 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 10 monclient: shutdown 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 msgr2=0x7f0f0011e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:51.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 0x7f0f0011e000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8007cf0 tx=0x7f0ef800e0e0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 shutdown_connections 2024-09-16T10:59:51.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f0009c2e0 0x7f0f0011e000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f0009bed0 0x7f0f00139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0ee0036f60 0x7f0ee0039420 unknown :-1 s=CLOSED pgs=3819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 --2- 172.21.15.5:0/1162470097 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f0009b3e0 0x7f0f001c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 >> 172.21.15.5:0/1162470097 conn(0x7f0f000a45d0 msgr2=0x7f0f0009e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 shutdown_connections 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 -- 172.21.15.5:0/1162470097 wait complete. 2024-09-16T10:59:51.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:51.657+0000 7f0f05749700 1 librados: shutdown 2024-09-16T10:59:54.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: get_monmap_and_config 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: build_initial_monmap 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: monmap: 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.799 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:54.798184+0000 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:54.798184+0000 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:54.800 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 Processor -- start 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 -- start start 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: init 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-a con 0x7f2d7c0a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d7c09f780 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:54.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d7c0a3510 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c0a3c40 con 0x7f2d7c09f780 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c1367d0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0a4300 con 0x7f2d7c0a2bf0 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d815f6700 10 monclient(hunting): authenticate will time out at 1453.374939s 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35452/0 (socket says 172.21.15.5:35452) 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1002748549 learned_addr learned my addr 172.21.15.5:0/1002748549 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request con 0x7f2d7c0a2bf0 auth_method 0 2024-09-16T10:59:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 msgr2=0x7f2d7c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 msgr2=0x7f2d7c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507180887 0 0) 0x7f2d7c1367d0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 10 monclient(hunting): _negotiate 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:54.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 10 monclient(hunting): my global_id is 64210 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.793+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680035d0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723361640 0 0) 0x7f2d680035d0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 msgr2=0x7f2d7c0a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 --2- 172.21.15.5:0/1002748549 >> v2:172.21.15.5:3300/0 conn(0x7f2d7c0a2bf0 0x7f2d7c0a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 >> v1:172.21.15.5:6789/0 conn(0x7f2d7c09f780 legacy=0x7f2d7c1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _finish_hunting 0 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 monclient: found mon.noname-c-legacy 2024-09-16T10:59:54.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c09fda0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _finish_auth 0 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _check_auth_tickets 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d815f6700 5 monclient: authenticate success, global_id 64210 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f2d740019f0 con 0x7f2d7c0a3510 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: dump: 2024-09-16T10:59:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T10:59:54.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 >> v1:172.21.15.143:6789/0 conn(0x7f2d7c0a3510 legacy=0x7f2d7c1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a20 0x7f2d68003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): picked mon.b con 0x7f2d68003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 0x7f2d68004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): picked mon.a con 0x7f2d68004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68005350 0x7f2d6800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): picked mon.c con 0x7f2d68005350 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0a4300 con 0x7f2d68004870 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6800c220 con 0x7f2d68003a20 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6800c3e0 con 0x7f2d68005350 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d815f6700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 0x7f2d68004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68005350 0x7f2d6800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): get_auth_request con 0x7f2d68004870 auth_method 0 2024-09-16T10:59:54.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): get_auth_request con 0x7f2d68005350 auth_method 0 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:54.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a20 0x7f2d68003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request con 0x7f2d68003a20 auth_method 0 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_done global_id 64210 payload 293 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 -- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68005350 msgr2=0x7f2d6800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68005350 0x7f2d6800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 -- 172.21.15.5:0/1002748549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a20 msgr2=0x7f2d68003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a20 0x7f2d68003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 monclient: found mon.a 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 -- 172.21.15.5:0/1002748549 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680035d0 con 0x7f2d68004870 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a20 0x7f2d68003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68005350 0x7f2d6800ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d80df5700 1 -- 172.21.15.5:0/1002748549 reap_dead start 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 0x7f2d68004cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d64007fd0 tx=0x7f2d680058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c3c0d7a445b04b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d640052d0 con 0x7f2d68004870 2024-09-16T10:59:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d7affd700 10 monclient: dump: 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.797+0000 7f2d815f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64005430 con 0x7f2d68004870 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 1 -- 172.21.15.5:0/1002748549 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d640055a0 con 0x7f2d68004870 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 4 set_mon_vals no callback set 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d7affd700 10 monclient: dump: 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:54.819 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 10 monclient: shutdown 2024-09-16T10:59:54.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:54.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 msgr2=0x7f2d68004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 0x7f2d68004cf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d64007fd0 tx=0x7f2d680058a0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- 172.21.15.5:0/1002748549 shutdown_connections 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1002748549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d68004870 0x7f2d68004cf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- 172.21.15.5:0/1002748549 >> 172.21.15.5:0/1002748549 conn(0x7f2d7c0a53e0 msgr2=0x7f2d7c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- 172.21.15.5:0/1002748549 shutdown_connections 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- 172.21.15.5:0/1002748549 wait complete. 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 10 monclient: build_initial_monmap 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 10 monclient: monmap: 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:54.827 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 librados: starting msgr at 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 librados: starting objecter 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 Processor -- start 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 -- start start 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 librados: setting wanted keys 2024-09-16T10:59:54.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 1 librados: calling monclient init 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.801+0000 7f2d815f6700 10 monclient: init 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 0x7f2d7c1ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-c con 0x7f2d68004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 0x7f2d7c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-b con 0x7f2d7c134fb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): picked mon.noname-a con 0x7f2d7c11e840 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0a7620 con 0x7f2d7c11e840 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1cb5e0 con 0x7f2d7c134fb0 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c120fb0 con 0x7f2d68004870 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient(hunting): authenticate will time out at 1453.384317s 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 0x7f2d7c1ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35470/0 (socket says 172.21.15.5:35470) 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 1 -- 172.21.15.5:0/1963047309 learned_addr learned my addr 172.21.15.5:0/1963047309 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request con 0x7f2d68004870 auth_method 0 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): get_auth_request con 0x7f2d7c11e840 auth_method 0 2024-09-16T10:59:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 0x7f2d7c1353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): get_auth_request con 0x7f2d7c134fb0 auth_method 0 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:54.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient(hunting): handle_auth_done global_id 54338 payload 824 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1963047309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 msgr2=0x7f2d7c1353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 0x7f2d7c1353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 msgr2=0x7f2d7c11eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient: _finish_hunting 0 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 monclient: found mon.noname-c 2024-09-16T10:59:54.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 -- 172.21.15.5:0/1963047309 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c09f4e0 con 0x7f2d68004870 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d80df5700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 0x7f2d7c1353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 client.?.objecter ms_handle_connect 0x7f2d68004870 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7bfff700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 0x7f2d7c1ca910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c010860 tx=0x7f2d6c010b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c015880 con 0x7f2d68004870 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: dump: 2024-09-16T10:59:54.834 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: _finish_auth 0 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:54.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c00c050 con 0x7f2d68004870 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 5 monclient: authenticate success, global_id 54338 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _renew_subs 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c1203e0 con 0x7f2d68004870 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c015c20 con 0x7f2d68004870 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 10 monclient: dump: 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:54.836 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 client.54338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d7a7fc700 4 set_mon_vals no callback set 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _renew_subs 2024-09-16T10:59:54.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c122460 con 0x7f2d68004870 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 librados: init done 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2d6c016420 con 0x7f2d68004870 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c0a7100 con 0x7f2d68004870 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 --2- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 0x7f2d5c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.805+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(398..398 src has 344..398) v4 ==== 5368+0+0 (secure 0 0 0) 0x7f2d6c048440 con 0x7f2d68004870 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d78ff9700 10 client.54338.objecter ms_dispatch 0x7f2d7c000fd0 osd_map(398..398 src has 344..398) v4 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d78ff9700 3 client.54338.objecter handle_osd_map got epochs [398,398] > 0 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d78ff9700 3 client.54338.objecter handle_osd_map decoding full epoch 398 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d78ff9700 20 client.54338.objecter dump_active .. 0 homeless 2024-09-16T10:59:54.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 0x7f2d5c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d7b7fe700 10 monclient: get_auth_request con 0x7f2d5c036f60 auth_method 0 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d78ff9700 10 client.54338.objecter ms_handle_connect 0x7f2d5c036f60 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.809+0000 7f2d7b7fe700 1 --2- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 0x7f2d5c039420 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7f2d64015040 tx=0x7f2d64009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.817+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2d6c027350 con 0x7f2d68004870 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.817+0000 7f2d78ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:54.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:54.817+0000 7f2d78ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d815f6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:59:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d815f6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d815f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2d7c0a2fd0 con 0x7f2d68004870 2024-09-16T10:59:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d78ff9700 1 -- 172.21.15.5:0/1963047309 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f2d6c01e3b0 con 0x7f2d68004870 2024-09-16T10:59:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d78ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.049+0000 7f2d78ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:55.056 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:59:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 10 librados: watch_flush enter 2024-09-16T10:59:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 10 librados: watch_flush exit 2024-09-16T10:59:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 20 client.54338.objecter shutdown clearing up homeless session... 2024-09-16T10:59:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 10 client.54338.objecter successfully canceled tick 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 msgr2=0x7f2d5c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 0x7f2d5c039420 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7f2d64015040 tx=0x7f2d64009030 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 10 monclient: shutdown 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 msgr2=0x7f2d7c1ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 0x7f2d7c1ca910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c010860 tx=0x7f2d6c010b70 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 shutdown_connections 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2d5c036f60 0x7f2d5c039420 unknown :-1 s=CLOSED pgs=3820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2d7c11e840 0x7f2d7c11eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c134fb0 0x7f2d7c1353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 --2- 172.21.15.5:0/1963047309 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2d68004870 0x7f2d7c1ca910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 >> 172.21.15.5:0/1963047309 conn(0x7f2d7c0a53e0 msgr2=0x7f2d7c1bdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 shutdown_connections 2024-09-16T10:59:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 -- 172.21.15.5:0/1963047309 wait complete. 2024-09-16T10:59:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.053+0000 7f2d815f6700 1 librados: shutdown 2024-09-16T10:59:55.070 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:59:55.070 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:59:55.070 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.177+0000 7f02a359e700 10 monclient: get_monmap_and_config 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.177+0000 7f02a359e700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.177+0000 7f02a359e700 10 monclient: monmap: 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:55.184448+0000 2024-09-16T10:59:55.185 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:55.184448+0000 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:55.186 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 Processor -- start 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- start start 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient: init 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f02a409e8e0 0x7f02a409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f02a409e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02a409d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 0x7f02a409f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f02a409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a4070e40 con 0x7f02a409d6a0 2024-09-16T10:59:55.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a40a01d0 con 0x7f02a409e8e0 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a409f740 con 0x7f02a409dfd0 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient(hunting): authenticate will time out at 1453.760794s 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 0x7f02a409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f0299d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f02a409d6a0 0x7f02a41336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46578/0 (socket says 172.21.15.5:46578) 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f0299d9b700 1 -- 172.21.15.5:0/638928112 learned_addr learned my addr 172.21.15.5:0/638928112 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): get_auth_request con 0x7f02a409dfd0 auth_method 0 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a259c700 1 --2- 172.21.15.5:0/638928112 >> v2:172.21.15.143:3300/0 conn(0x7f02a409e8e0 0x7f02a409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a259c700 10 monclient(hunting): get_auth_request con 0x7f02a409e8e0 auth_method 0 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_done global_id 54341 payload 274 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 -- 172.21.15.5:0/638928112 >> v2:172.21.15.143:3300/0 conn(0x7f02a409e8e0 msgr2=0x7f02a409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 --2- 172.21.15.5:0/638928112 >> v2:172.21.15.143:3300/0 conn(0x7f02a409e8e0 0x7f02a409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 -- 172.21.15.5:0/638928112 >> v1:172.21.15.5:6789/0 conn(0x7f02a409d6a0 legacy=0x7f02a41336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 monclient: found mon.noname-b 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 -- 172.21.15.5:0/638928112 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a409fef0 con 0x7f02a409dfd0 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a2d9d700 1 --2- 172.21.15.5:0/638928112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 0x7f02a409f200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f029c009dc0 tx=0x7f029c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ddc4a38dbd4980 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 1 -- 172.21.15.5:0/638928112 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c00e070 con 0x7f02a409dfd0 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: dump: 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.193 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 1 -- 172.21.15.5:0/638928112 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c00bb90 con 0x7f02a409dfd0 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 5 monclient: authenticate success, global_id 54341 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:55.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 1 -- 172.21.15.5:0/638928112 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c012430 con 0x7f02a409dfd0 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 4 set_mon_vals no callback set 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a1d9b700 10 monclient: dump: 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.195 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 10 monclient: shutdown 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- 172.21.15.5:0/638928112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 msgr2=0x7f02a409f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 --2- 172.21.15.5:0/638928112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 0x7f02a409f200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f029c009dc0 tx=0x7f029c00b330 comp rx=0 tx=0).stop 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- 172.21.15.5:0/638928112 shutdown_connections 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 --2- 172.21.15.5:0/638928112 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409dfd0 0x7f02a409f200 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 --2- 172.21.15.5:0/638928112 >> v2:172.21.15.143:3300/0 conn(0x7f02a409e8e0 0x7f02a409ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- 172.21.15.5:0/638928112 >> 172.21.15.5:0/638928112 conn(0x7f02a40a45d0 msgr2=0x7f02a40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- 172.21.15.5:0/638928112 shutdown_connections 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.181+0000 7f02a359e700 1 -- 172.21.15.5:0/638928112 wait complete. 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: monmap: 2024-09-16T10:59:55.196 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 librados: starting msgr at 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 librados: starting objecter 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 Processor -- start 2024-09-16T10:59:55.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- start start 2024-09-16T10:59:55.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 librados: setting wanted keys 2024-09-16T10:59:55.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 librados: calling monclient init 2024-09-16T10:59:55.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: init 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a409d6a0 0x7f02a4132f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f02a409d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 0x7f02a4133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-a con 0x7f02a409dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409e8e0 0x7f02a406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f02a409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a409b0d0 con 0x7f02a409dfd0 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a41339c0 con 0x7f02a409d6a0 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a411f1e0 con 0x7f02a409e8e0 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): authenticate will time out at 1453.765430s 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 0x7f02a4133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 0x7f02a4133480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35480/0 (socket says 172.21.15.5:35480) 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 -- 172.21.15.5:0/2896418071 learned_addr learned my addr 172.21.15.5:0/2896418071 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): get_auth_request con 0x7f02a409dfd0 auth_method 0 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409e8e0 0x7f02a406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a409d6a0 0x7f02a4132f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): get_auth_request con 0x7f02a409d6a0 auth_method 0 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): get_auth_request con 0x7f02a409e8e0 auth_method 0 2024-09-16T10:59:55.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient(hunting): handle_auth_done global_id 74103 payload 824 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409e8e0 msgr2=0x7f02a406e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f02a409e8e0 0x7f02a406e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a409d6a0 msgr2=0x7f02a4132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02a409d6a0 0x7f02a4132f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 monclient: found mon.noname-a 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c009a70 con 0x7f02a409dfd0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f0299d9b700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 0x7f02a4133480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c700 tx=0x7f028c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 client.?.objecter ms_handle_connect 0x7f02a409dfd0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c005d90 con 0x7f02a409dfd0 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: dump: 2024-09-16T10:59:55.207 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c0070d0 con 0x7f02a409dfd0 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 5 monclient: authenticate success, global_id 74103 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _renew_subs 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a40a3980 con 0x7f02a409dfd0 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c007440 con 0x7f02a409dfd0 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 10 monclient: dump: 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.209 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a159a700 4 set_mon_vals no callback set 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 client.74103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _renew_subs 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a4120340 con 0x7f02a409dfd0 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 librados: init done 2024-09-16T10:59:55.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f028c0075a0 con 0x7f02a409dfd0 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 0x7f02a406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.185+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4067380 con 0x7f02a41ca580 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 --2- 172.21.15.5:0/2896418071 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0284036f60 0x7f0284039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 0x7f02a406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(398..398 src has 344..398) v4 ==== 5368+0+0 (secure 0 0 0) 0x7f028c047260 con 0x7f02a409dfd0 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient: get_auth_request con 0x7f02a41ca580 auth_method 0 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 client.74103.objecter ms_dispatch 0x7f02a4000fd0 osd_map(398..398 src has 344..398) v4 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 3 client.74103.objecter handle_osd_map got epochs [398,398] > 0 2024-09-16T10:59:55.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 3 client.74103.objecter handle_osd_map decoding full epoch 398 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 20 client.74103.objecter dump_active .. 0 homeless 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 10 monclient(hunting): handle_auth_done global_id 74106 payload 824 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a2d9d700 1 --2- 172.21.15.5:0/2896418071 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0284036f60 0x7f0284039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02a4120530 con 0x7f02a41ca580 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a2d9d700 10 monclient: get_auth_request con 0x7f0284036f60 auth_method 0 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 client.74103.objecter ms_handle_connect 0x7f02a41ca580 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 client.74103.objecter resend_mon_ops 2024-09-16T10:59:55.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a259c700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 0x7f02a406ebf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294004550 con 0x7f02a41ca580 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 monclient: dump: 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.213 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 client.74103.objecter ms_handle_connect 0x7f0284036f60 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f02a2d9d700 1 --2- 172.21.15.5:0/2896418071 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0284036f60 0x7f0284039420 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f029c009dc0 tx=0x7f029c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f02940036c0 con 0x7f02a41ca580 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 msgr2=0x7f02a406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 0x7f02a406ebf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294011bf0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 msgr2=0x7f02a406ebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.189+0000 7f029b7fe700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a41ca580 0x7f02a406ebf0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294011bf0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:55.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 0x7f02a41bbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a406f540 con 0x7f02a406f130 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 0x7f02a41bbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient: get_auth_request con 0x7f02a406f130 auth_method 0 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 10 monclient(hunting): handle_auth_done global_id 74109 payload 824 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 1 -- 172.21.15.5:0/2896418071 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02a4133d50 con 0x7f02a406f130 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a259c700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 0x7f02a41bbaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 client.74103.objecter ms_handle_connect 0x7f02a406f130 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 client.74103.objecter resend_mon_ops 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294025070 con 0x7f02a406f130 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 monclient: dump: 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.218 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f02940154c0 con 0x7f02a406f130 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 msgr2=0x7f02a41bbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 0x7f02a41bbaa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294004770 comp rx=0 tx=0).stop 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 msgr2=0x7f02a41bbaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f029b7fe700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a406f130 0x7f02a41bbaa0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0294011830 tx=0x7f0294004770 comp rx=0 tx=0).stop 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T10:59:55.220 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.221 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.222 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:55.223 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.224 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:55.225 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.226 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.227 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.228 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.229 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.230 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.231 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:59:55.232 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.232 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.232 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.232 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:55.232 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:55.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 librados: watch_flush enter 2024-09-16T10:59:55.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 librados: watch_flush exit 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 20 client.74103.objecter shutdown clearing up homeless session... 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 client.74103.objecter successfully canceled tick 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0284036f60 msgr2=0x7f0284039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 --2- 172.21.15.5:0/2896418071 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0284036f60 0x7f0284039420 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f029c009dc0 tx=0x7f029c011000 comp rx=0 tx=0).stop 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 10 monclient: shutdown 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 msgr2=0x7f02a4133480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 --2- 172.21.15.5:0/2896418071 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02a409dfd0 0x7f02a4133480 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f028c00c700 tx=0x7f028c00ca10 comp rx=0 tx=0).stop 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a2d9d700 1 -- 172.21.15.5:0/2896418071 reap_dead start 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a2d9d700 1 -- 172.21.15.5:0/2896418071 reap_dead start 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 shutdown_connections 2024-09-16T10:59:55.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 >> 172.21.15.5:0/2896418071 conn(0x7f02a40a45d0 msgr2=0x7f02a409b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 shutdown_connections 2024-09-16T10:59:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 -- 172.21.15.5:0/2896418071 wait complete. 2024-09-16T10:59:55.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.201+0000 7f02a359e700 1 librados: shutdown 2024-09-16T10:59:55.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:55.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:59:55.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:59:55.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: get_monmap_and_config 2024-09-16T10:59:55.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: monmap: 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:55.353662+0000 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:55.353662+0000 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:55.355 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:55.356 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 Processor -- start 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 -- start start 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: init 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f723409b3e0 0x7f723409b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-a con 0x7f723409b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:55.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f723409bd00 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f723409c110 0x7f7234132dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-c con 0x7f723409c110 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723409c560 con 0x7f723409bd00 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723409e350 con 0x7f723409b3e0 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723409ca20 con 0x7f723409c110 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7238afc700 10 monclient(hunting): authenticate will time out at 1453.930448s 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f723409b3e0 0x7f723409b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f722affd700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f723409bd00 0x7f72341326b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46582/0 (socket says 172.21.15.5:46582) 2024-09-16T10:59:55.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f722affd700 1 -- 172.21.15.5:0/1532169522 learned_addr learned my addr 172.21.15.5:0/1532169522 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): get_auth_request con 0x7f723409b3e0 auth_method 0 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 1 --2- 172.21.15.5:0/1532169522 >> v2:172.21.15.143:3300/0 conn(0x7f723409c110 0x7f7234132dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): get_auth_request con 0x7f723409c110 auth_method 0 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient(hunting): handle_auth_done global_id 74112 payload 274 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 -- 172.21.15.5:0/1532169522 >> v2:172.21.15.143:3300/0 conn(0x7f723409c110 msgr2=0x7f7234132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 --2- 172.21.15.5:0/1532169522 >> v2:172.21.15.143:3300/0 conn(0x7f723409c110 0x7f7234132dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 -- 172.21.15.5:0/1532169522 >> v1:172.21.15.5:6789/0 conn(0x7f723409bd00 legacy=0x7f72341326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7232ffd700 1 -- 172.21.15.5:0/1532169522 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758755501 0 0) 0x7f723409c560 con 0x7f723409bd00 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 monclient: found mon.noname-a 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 -- 172.21.15.5:0/1532169522 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723409cd00 con 0x7f723409b3e0 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f7232ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.349+0000 7f72337fe700 1 --2- 172.21.15.5:0/1532169522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723409b7c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f721c009850 tx=0x7f721c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b59f84c123f40d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 1 -- 172.21.15.5:0/1532169522 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c011830 con 0x7f723409b3e0 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: dump: 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.367 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 5 monclient: authenticate success, global_id 74112 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 1 -- 172.21.15.5:0/1532169522 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721c011990 con 0x7f723409b3e0 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 4 set_mon_vals no callback set 2024-09-16T10:59:55.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 1 -- 172.21.15.5:0/1532169522 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721c011d20 con 0x7f723409b3e0 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7232ffd700 10 monclient: dump: 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.369 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: shutdown 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- 172.21.15.5:0/1532169522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 msgr2=0x7f723409b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- 172.21.15.5:0/1532169522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723409b7c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f721c009850 tx=0x7f721c009b60 comp rx=0 tx=0).stop 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- 172.21.15.5:0/1532169522 shutdown_connections 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- 172.21.15.5:0/1532169522 >> v2:172.21.15.143:3300/0 conn(0x7f723409c110 0x7f7234132dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- 172.21.15.5:0/1532169522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723409b7c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- 172.21.15.5:0/1532169522 >> 172.21.15.5:0/1532169522 conn(0x7f72340a45d0 msgr2=0x7f72340a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- 172.21.15.5:0/1532169522 shutdown_connections 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- 172.21.15.5:0/1532169522 wait complete. 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: monmap: 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 librados: starting msgr at 2024-09-16T10:59:55.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 librados: starting objecter 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 Processor -- start 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- start start 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 librados: setting wanted keys 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 librados: calling monclient init 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: init 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723411efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-a con 0x7f723409b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 0x7f723411f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-c con 0x7f723409bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f723409c110 0x7f723411fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): picked mon.noname-b con 0x7f723409c110 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234122ca0 con 0x7f723409b3e0 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234070770 con 0x7f723409c110 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72340a3740 con 0x7f723409bd00 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f7238afc700 10 monclient(hunting): authenticate will time out at 1453.935788s 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723411efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f722affd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 0x7f723411f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723411efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35518/0 (socket says 172.21.15.5:35518) 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 1 -- 172.21.15.5:0/83714928 learned_addr learned my addr 172.21.15.5:0/83714928 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:55.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 10 monclient(hunting): get_auth_request con 0x7f723409b3e0 auth_method 0 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f72337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f722affd700 10 monclient(hunting): get_auth_request con 0x7f723409bd00 auth_method 0 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f722affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f722affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.353+0000 7f722affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f723409c110 0x7f723411fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): get_auth_request con 0x7f723409c110 auth_method 0 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient(hunting): handle_auth_done global_id 54347 payload 824 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f723409c110 msgr2=0x7f723411fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f723409c110 0x7f723411fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 msgr2=0x7f723411efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f723409b3e0 0x7f723411efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 monclient: found mon.noname-c 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c0094b0 con 0x7f723409bd00 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f722affd700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 0x7f723411f4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72240011e0 tx=0x7f722400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 client.?.objecter ms_handle_connect 0x7f723409bd00 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224004ca0 con 0x7f723409bd00 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: dump: 2024-09-16T10:59:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 5 monclient: authenticate success, global_id 54347 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 monclient: _renew_subs 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224004e00 con 0x7f723409bd00 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723409b0d0 con 0x7f723409bd00 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224007310 con 0x7f723409bd00 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 client.54347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 monclient: dump: 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.378 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 monclient: _renew_subs 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72327fc700 4 set_mon_vals no callback set 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7234121290 con 0x7f723409bd00 2024-09-16T10:59:55.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 1 librados: init done 2024-09-16T10:59:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7224007470 con 0x7f723409bd00 2024-09-16T10:59:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 --2- 172.21.15.5:0/83714928 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7218036f90 0x7f7218039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(398..398 src has 344..398) v4 ==== 5368+0+0 (secure 0 0 0) 0x7f7224047950 con 0x7f723409bd00 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 10 client.54347.objecter ms_dispatch 0x7f7234000fd0 osd_map(398..398 src has 344..398) v4 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 3 client.54347.objecter handle_osd_map got epochs [398,398] > 0 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 3 client.54347.objecter handle_osd_map decoding full epoch 398 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 0x7f723411b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7234067380 con 0x7f723411b540 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7230ff9700 20 client.54347.objecter dump_active .. 0 homeless 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72337fe700 1 --2- 172.21.15.5:0/83714928 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7218036f90 0x7f7218039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 0x7f723411b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f72337fe700 10 monclient: get_auth_request con 0x7f7218036f90 auth_method 0 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient: get_auth_request con 0x7f723411b540 auth_method 0 2024-09-16T10:59:55.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.357+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7233fff700 10 monclient(hunting): handle_auth_done global_id 54350 payload 824 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7233fff700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f723409ff00 con 0x7f723411b540 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f72337fe700 1 --2- 172.21.15.5:0/83714928 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7218036f90 0x7f7218039450 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7f721c009ba0 tx=0x7f721c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 client.54347.objecter ms_handle_connect 0x7f7218036f90 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 client.54347.objecter ms_handle_connect 0x7f723411b540 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 client.54347.objecter resend_mon_ops 2024-09-16T10:59:55.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7233fff700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 0x7f723411b920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c00f490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c00e040 con 0x7f723411b540 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 monclient: dump: 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f722c00f040 con 0x7f723411b540 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 msgr2=0x7f723411b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 0x7f723411b920 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c00f490 comp rx=0 tx=0).stop 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 msgr2=0x7f723411b920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.361+0000 7f7230ff9700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411b540 0x7f723411b920 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c00f490 comp rx=0 tx=0).stop 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 0x7f72341d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72341224b0 con 0x7f723411be60 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 0x7f72341d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient: get_auth_request con 0x7f723411be60 auth_method 0 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 10 monclient(hunting): handle_auth_done global_id 54353 payload 824 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 1 -- 172.21.15.5:0/83714928 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f723409b7c0 con 0x7f723411be60 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 0x7f72341d28f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 client.54347.objecter ms_handle_connect 0x7f723411be60 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 client.54347.objecter resend_mon_ops 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c023070 con 0x7f723411be60 2024-09-16T10:59:55.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 monclient: dump: 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f722c015640 con 0x7f723411be60 2024-09-16T10:59:55.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 msgr2=0x7f72341d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 0x7f72341d28f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c004af0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 msgr2=0x7f72341d28f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7230ff9700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723411be60 0x7f72341d28f0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f722c009e30 tx=0x7f722c004af0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:55.390 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 1, 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.391 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.392 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:55.393 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:55.394 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.395 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:55.396 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.397 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.398 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.399 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.400 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:55.401 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.402 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:55.403 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 librados: watch_flush enter 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 librados: watch_flush exit 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 20 client.54347.objecter shutdown clearing up homeless session... 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 client.54347.objecter successfully canceled tick 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7218036f90 msgr2=0x7f7218039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 --2- 172.21.15.5:0/83714928 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7218036f90 0x7f7218039450 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7f721c009ba0 tx=0x7f721c008030 comp rx=0 tx=0).stop 2024-09-16T10:59:55.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 10 monclient: shutdown 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7233fff700 1 -- 172.21.15.5:0/83714928 reap_dead start 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 msgr2=0x7f723411f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 0x7f723411f4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72240011e0 tx=0x7f722400c800 comp rx=0 tx=0).stop 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 shutdown_connections 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 --2- 172.21.15.5:0/83714928 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f723409bd00 0x7f723411f4f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 >> 172.21.15.5:0/83714928 conn(0x7f72340a45d0 msgr2=0x7f723406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 shutdown_connections 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 -- 172.21.15.5:0/83714928 wait complete. 2024-09-16T10:59:55.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.369+0000 7f7238afc700 1 librados: shutdown 2024-09-16T10:59:55.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:55.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:59:55.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.517+0000 7f313f159700 10 monclient: get_monmap_and_config 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.517+0000 7f313f159700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.517+0000 7f313f159700 10 monclient: monmap: 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:55.524375+0000 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:55.524375+0000 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:55.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 Processor -- start 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 -- start start 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient: init 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f31380a0680 0x7f31380a1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): picked mon.noname-a con 0x7f31380a0680 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f313809cd70 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3138137930 0x7f3138139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): picked mon.noname-c con 0x7f3138137930 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313809c0b0 con 0x7f313809cd70 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380a0150 con 0x7f31380a0680 2024-09-16T10:59:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f313809c390 con 0x7f3138137930 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient(hunting): authenticate will time out at 1454.101112s 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f31380a0680 0x7f31380a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 0x7f313809ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46598/0 (socket says 172.21.15.5:46598) 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 -- 172.21.15.5:0/4107924562 learned_addr learned my addr 172.21.15.5:0/4107924562 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): get_auth_request con 0x7f31380a0680 auth_method 0 2024-09-16T10:59:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313d6f6700 1 --2- 172.21.15.5:0/4107924562 >> v2:172.21.15.143:3300/0 conn(0x7f3138137930 0x7f3138139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 -- 172.21.15.5:0/4107924562 >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 legacy=0x7f313809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 -- 172.21.15.5:0/4107924562 >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 legacy=0x7f313809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 --1- 172.21.15.5:0/4107924562 >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 0x7f313809ea30 :-1 s=OPENED pgs=35 cs=1 l=1).handle_message read tag failed 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f3137fff700 1 --1- 172.21.15.5:0/4107924562 >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 0x7f313809ea30 :-1 s=OPENED pgs=35 cs=1 l=1).fault on lossy channel, failing 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.5:6789/0 2024-09-16T10:59:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313d6f6700 10 monclient(hunting): get_auth_request con 0x7f3138137930 auth_method 0 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313d6f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313d6f6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313d6f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient(hunting): handle_auth_done global_id 74121 payload 274 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 -- 172.21.15.5:0/4107924562 >> v2:172.21.15.143:3300/0 conn(0x7f3138137930 msgr2=0x7f3138139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 --2- 172.21.15.5:0/4107924562 >> v2:172.21.15.143:3300/0 conn(0x7f3138137930 0x7f3138139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 -- 172.21.15.5:0/4107924562 >> v1:172.21.15.5:6789/0 conn(0x7f313809cd70 legacy=0x7f313809ea30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 monclient: found mon.noname-a 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 -- 172.21.15.5:0/4107924562 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313809c670 con 0x7f31380a0680 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313cef5700 1 --2- 172.21.15.5:0/4107924562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31380a0680 0x7f31380a1650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f313806b9a0 tx=0x7f312c009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1af49156f1498b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 1 -- 172.21.15.5:0/4107924562 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c004230 con 0x7f31380a0680 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: dump: 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 5 monclient: authenticate success, global_id 74121 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 1 -- 172.21.15.5:0/4107924562 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c004390 con 0x7f31380a0680 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 4 set_mon_vals no callback set 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 1 -- 172.21.15.5:0/4107924562 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c0036c0 con 0x7f31380a0680 2024-09-16T10:59:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f31377fe700 10 monclient: dump: 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.536 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 10 monclient: shutdown 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 -- 172.21.15.5:0/4107924562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31380a0680 msgr2=0x7f31380a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.521+0000 7f313f159700 1 --2- 172.21.15.5:0/4107924562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31380a0680 0x7f31380a1650 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f313806b9a0 tx=0x7f312c009ab0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- 172.21.15.5:0/4107924562 shutdown_connections 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 --2- 172.21.15.5:0/4107924562 >> v2:172.21.15.143:3300/0 conn(0x7f3138137930 0x7f3138139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 --2- 172.21.15.5:0/4107924562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f31380a0680 0x7f31380a1650 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- 172.21.15.5:0/4107924562 >> 172.21.15.5:0/4107924562 conn(0x7f31380a0b30 msgr2=0x7f31380a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- 172.21.15.5:0/4107924562 shutdown_connections 2024-09-16T10:59:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- 172.21.15.5:0/4107924562 wait complete. 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient: monmap: 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 librados: starting msgr at 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 librados: starting objecter 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 Processor -- start 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- start start 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 librados: setting wanted keys 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 librados: calling monclient init 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient: init 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 0x7f3138136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): picked mon.noname-c con 0x7f313809cd70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380a0680 0x7f3138130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): picked mon.noname-b con 0x7f31380a0680 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 0x7f313813b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): picked mon.noname-a con 0x7f3138137930 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313811b5c0 con 0x7f3138137930 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313811b810 con 0x7f31380a0680 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380a1fc0 con 0x7f313809cd70 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313f159700 10 monclient(hunting): authenticate will time out at 1454.107773s 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313cef5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 0x7f3138136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.525+0000 7f313d6f6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 0x7f313813b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 0x7f313813b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35534/0 (socket says 172.21.15.5:35534) 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 1 -- 172.21.15.5:0/430526370 learned_addr learned my addr 172.21.15.5:0/430526370 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): get_auth_request con 0x7f3138137930 auth_method 0 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): get_auth_request con 0x7f313809cd70 auth_method 0 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380a0680 0x7f3138130e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): get_auth_request con 0x7f31380a0680 auth_method 0 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient(hunting): handle_auth_done global_id 54356 payload 824 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380a0680 msgr2=0x7f3138130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31380a0680 0x7f3138130e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 msgr2=0x7f313813b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 0x7f313813b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 monclient: found mon.noname-c 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c0094b0 con 0x7f313809cd70 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313d6f6700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3138137930 0x7f313813b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313cef5700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 0x7f3138136cd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f313806b190 tx=0x7f312c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 client.?.objecter ms_handle_connect 0x7f313809cd70 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c006ea0 con 0x7f313809cd70 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: dump: 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c01f070 con 0x7f313809cd70 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 5 monclient: authenticate success, global_id 54356 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 monclient: _renew_subs 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313809b210 con 0x7f313809cd70 2024-09-16T10:59:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c003ea0 con 0x7f313809cd70 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 monclient: dump: 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.551 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3136ffd700 4 set_mon_vals no callback set 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 client.54356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 monclient: _renew_subs 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313813c310 con 0x7f313809cd70 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 1 librados: init done 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 0x7f313813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313811b9a0 con 0x7f313813b7b0 2024-09-16T10:59:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 0x7f313813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient: get_auth_request con 0x7f313813b7b0 auth_method 0 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f312c021030 con 0x7f313809cd70 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 --2- 172.21.15.5:0/430526370 >> v2:172.21.15.143:6812/1098205034 conn(0x7f311c036f60 0x7f311c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(398..398 src has 344..398) v4 ==== 5368+0+0 (secure 0 0 0) 0x7f312c049090 con 0x7f313809cd70 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 10 monclient(hunting): handle_auth_done global_id 64234 payload 824 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 10 client.54356.objecter ms_dispatch 0x7f3138000fd0 osd_map(398..398 src has 344..398) v4 2024-09-16T10:59:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f3137fff700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f313813c0d0 con 0x7f313813b7b0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 3 client.54356.objecter handle_osd_map got epochs [398,398] > 0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.529+0000 7f31357fa700 3 client.54356.objecter handle_osd_map decoding full epoch 398 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f313d6f6700 1 --2- 172.21.15.5:0/430526370 >> v2:172.21.15.143:6812/1098205034 conn(0x7f311c036f60 0x7f311c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f313d6f6700 10 monclient: get_auth_request con 0x7f311c036f60 auth_method 0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 20 client.54356.objecter dump_active .. 0 homeless 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f3137fff700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 0x7f313813bb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3124000f60 tx=0x7f312400fb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 client.54356.objecter ms_handle_connect 0x7f313813b7b0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 client.54356.objecter resend_mon_ops 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124005b70 con 0x7f313813b7b0 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 monclient: dump: 2024-09-16T10:59:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 client.54356.objecter ms_handle_connect 0x7f311c036f60 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f313d6f6700 1 --2- 172.21.15.5:0/430526370 >> v2:172.21.15.143:6812/1098205034 conn(0x7f311c036f60 0x7f311c039420 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f31300094c0 tx=0x7f31300098b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f3124005cd0 con 0x7f313813b7b0 2024-09-16T10:59:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 msgr2=0x7f313813bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 0x7f313813bb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3124000f60 tx=0x7f312400fb00 comp rx=0 tx=0).stop 2024-09-16T10:59:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 msgr2=0x7f313813bb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.533+0000 7f31357fa700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313813b7b0 0x7f313813bb90 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3124000f60 tx=0x7f312400fb00 comp rx=0 tx=0).stop 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f313f159700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f313f159700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 0x7f313806e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f313f159700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138067580 con 0x7f313806e330 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 0x7f313806e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient: get_auth_request con 0x7f313806e330 auth_method 0 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 10 monclient(hunting): handle_auth_done global_id 64237 payload 824 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 1 -- 172.21.15.5:0/430526370 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3138071830 con 0x7f313806e330 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 client.54356.objecter ms_handle_connect 0x7f313806e330 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 client.54356.objecter resend_mon_ops 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f3137fff700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 0x7f313806e740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3124017000 tx=0x7f31240058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124023070 con 0x7f313806e330 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 monclient: dump: 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.559 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f3124007730 con 0x7f313806e330 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 msgr2=0x7f313806e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 0x7f313806e740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3124017000 tx=0x7f31240058a0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 msgr2=0x7f313806e740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.545+0000 7f31357fa700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f313806e330 0x7f313806e740 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3124017000 tx=0x7f31240058a0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 1, 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.568 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.569 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.572 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:55.573 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:55.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:55.574 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:55.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:55.574 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:55.574 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:55.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 10 librados: watch_flush enter 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 10 librados: watch_flush exit 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 20 client.54356.objecter shutdown clearing up homeless session... 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 10 client.54356.objecter successfully canceled tick 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 >> v2:172.21.15.143:6812/1098205034 conn(0x7f311c036f60 msgr2=0x7f311c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 --2- 172.21.15.5:0/430526370 >> v2:172.21.15.143:6812/1098205034 conn(0x7f311c036f60 0x7f311c039420 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f31300094c0 tx=0x7f31300098b0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 10 monclient: shutdown 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313d6f6700 1 -- 172.21.15.5:0/430526370 reap_dead start 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 msgr2=0x7f3138136cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 0x7f3138136cd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f313806b190 tx=0x7f312c004770 comp rx=0 tx=0).stop 2024-09-16T10:59:55.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 shutdown_connections 2024-09-16T10:59:55.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 --2- 172.21.15.5:0/430526370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f313809cd70 0x7f3138136cd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 >> 172.21.15.5:0/430526370 conn(0x7f31380a0b30 msgr2=0x7f313809e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 shutdown_connections 2024-09-16T10:59:55.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 -- 172.21.15.5:0/430526370 wait complete. 2024-09-16T10:59:55.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.549+0000 7f313f159700 1 librados: shutdown 2024-09-16T10:59:55.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:55.577 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T10:59:55.577 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T10:59:55.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: get_monmap_and_config 2024-09-16T10:59:55.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: monmap: 2024-09-16T10:59:55.700 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.700 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.700 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:55.698722+0000 2024-09-16T10:59:55.701 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:55.698722+0000 2024-09-16T10:59:55.701 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.701 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.701 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:55.701 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:55.702 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:55.702 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:55.702 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:55.702 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 Processor -- start 2024-09-16T10:59:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 -- start start 2024-09-16T10:59:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: init 2024-09-16T10:59:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35d009e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f35d009f3d0 0x7f35d013f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-a con 0x7f35d009f3d0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35d013f710 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d009fb80 con 0x7f35d013f710 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d009b970 con 0x7f35d009e8e0 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d009c030 con 0x7f35d009f3d0 2024-09-16T10:59:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d65f1700 10 monclient(hunting): authenticate will time out at 1454.275103s 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f35d009f3d0 0x7f35d013f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f35d009f3d0 0x7f35d013f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35538/0 (socket says 172.21.15.5:35538) 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 -- 172.21.15.5:0/3815053878 learned_addr learned my addr 172.21.15.5:0/3815053878 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): get_auth_request con 0x7f35d009f3d0 auth_method 0 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient(hunting): handle_auth_done global_id 74127 payload 274 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 -- 172.21.15.5:0/3815053878 >> v1:172.21.15.143:6789/0 conn(0x7f35d009e8e0 legacy=0x7f35d009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 -- 172.21.15.5:0/3815053878 >> v1:172.21.15.5:6789/0 conn(0x7f35d013f710 legacy=0x7f35d0141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 monclient: found mon.noname-a 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 -- 172.21.15.5:0/3815053878 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d009bc50 con 0x7f35d009f3d0 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.693+0000 7f35d4dee700 1 --2- 172.21.15.5:0/3815053878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d013f1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009e50 tx=0x7f35cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862b40006a4316d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 1 -- 172.21.15.5:0/3815053878 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc00e070 con 0x7f35d009f3d0 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: dump: 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 5 monclient: authenticate success, global_id 74127 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 1 -- 172.21.15.5:0/3815053878 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc003a40 con 0x7f35d009f3d0 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 4 set_mon_vals no callback set 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 1 -- 172.21.15.5:0/3815053878 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc003df0 con 0x7f35d009f3d0 2024-09-16T10:59:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35c7fff700 10 monclient: dump: 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: shutdown 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- 172.21.15.5:0/3815053878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 msgr2=0x7f35d013f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 --2- 172.21.15.5:0/3815053878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d013f1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009e50 tx=0x7f35cc00b330 comp rx=0 tx=0).stop 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- 172.21.15.5:0/3815053878 shutdown_connections 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 --2- 172.21.15.5:0/3815053878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d013f1d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- 172.21.15.5:0/3815053878 >> 172.21.15.5:0/3815053878 conn(0x7f35d00a45d0 msgr2=0x7f35d00a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- 172.21.15.5:0/3815053878 shutdown_connections 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- 172.21.15.5:0/3815053878 wait complete. 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: build_initial_monmap 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: monmap: 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:55.726 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 librados: starting msgr at 2024-09-16T10:59:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 librados: starting objecter 2024-09-16T10:59:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 Processor -- start 2024-09-16T10:59:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- start start 2024-09-16T10:59:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 librados: setting wanted keys 2024-09-16T10:59:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 librados: calling monclient init 2024-09-16T10:59:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: init 2024-09-16T10:59:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d009e8e0 0x7f35d0138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-b con 0x7f35d009e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-a con 0x7f35d009f3d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f35d013f710 0x7f35d006fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): picked mon.noname-c con 0x7f35d013f710 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d01250a0 con 0x7f35d009f3d0 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d01252f0 con 0x7f35d009e8e0 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d01cfa40 con 0x7f35d013f710 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d65f1700 10 monclient(hunting): authenticate will time out at 1454.279785s 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.697+0000 7f35d4dee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f35d013f710 0x7f35d006fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35552/0 (socket says 172.21.15.5:35552) 2024-09-16T10:59:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 -- 172.21.15.5:0/2210993170 learned_addr learned my addr 172.21.15.5:0/2210993170 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): get_auth_request con 0x7f35d009f3d0 auth_method 0 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): get_auth_request con 0x7f35d013f710 auth_method 0 2024-09-16T10:59:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d009e8e0 0x7f35d0138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): get_auth_request con 0x7f35d009e8e0 auth_method 0 2024-09-16T10:59:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d55ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d5df0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient(hunting): handle_auth_done global_id 74130 payload 824 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 -- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f35d013f710 msgr2=0x7f35d006fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f35d013f710 0x7f35d006fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 -- 172.21.15.5:0/2210993170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d009e8e0 msgr2=0x7f35d0138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d009e8e0 0x7f35d0138920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient: _finish_hunting 0 2024-09-16T10:59:55.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 monclient: found mon.noname-a 2024-09-16T10:59:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 -- 172.21.15.5:0/2210993170 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc009b30 con 0x7f35d009f3d0 2024-09-16T10:59:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d4dee700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35cc015040 tx=0x7f35cc004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 client.?.objecter ms_handle_connect 0x7f35d009f3d0 2024-09-16T10:59:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc004440 con 0x7f35d009f3d0 2024-09-16T10:59:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: dump: 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.739 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: _finish_auth 0 2024-09-16T10:59:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:59:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc0045a0 con 0x7f35d009f3d0 2024-09-16T10:59:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 5 monclient: authenticate success, global_id 74130 2024-09-16T10:59:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _renew_subs 2024-09-16T10:59:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00a2ff0 con 0x7f35d009f3d0 2024-09-16T10:59:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 client.74130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _renew_subs 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc004950 con 0x7f35d009f3d0 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d0124a50 con 0x7f35d009f3d0 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 1 librados: init done 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c6ffd700 4 set_mon_vals no callback set 2024-09-16T10:59:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 10 monclient: dump: 2024-09-16T10:59:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d009ecc0 con 0x7f35d009f3d0 2024-09-16T10:59:55.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f35cc012430 con 0x7f35d009f3d0 2024-09-16T10:59:55.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 --2- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 0x7f35b4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:55.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.701+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(398..398 src has 344..398) v4 ==== 5368+0+0 (secure 0 0 0) 0x7f35cc049210 con 0x7f35d009f3d0 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35c5ffb700 10 client.74130.objecter ms_dispatch 0x7f35d0000fd0 osd_map(398..398 src has 344..398) v4 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35c5ffb700 3 client.74130.objecter handle_osd_map got epochs [398,398] > 0 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35c5ffb700 3 client.74130.objecter handle_osd_map decoding full epoch 398 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35c5ffb700 20 client.74130.objecter dump_active .. 0 homeless 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35d55ef700 1 --2- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 0x7f35b4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35d55ef700 10 monclient: get_auth_request con 0x7f35b4036f60 auth_method 0 2024-09-16T10:59:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35c5ffb700 10 client.74130.objecter ms_handle_connect 0x7f35b4036f60 2024-09-16T10:59:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.705+0000 7f35d55ef700 1 --2- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 0x7f35b4039420 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005150 tx=0x7f35c0007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.713+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f35cc024d00 con 0x7f35d009f3d0 2024-09-16T10:59:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.713+0000 7f35c5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.713+0000 7f35c5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.945+0000 7f35d65f1700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T10:59:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.945+0000 7f35d65f1700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T10:59:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.945+0000 7f35d65f1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:55.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.945+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f35d0125430 con 0x7f35d009f3d0 2024-09-16T10:59:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.949+0000 7f35c5ffb700 1 -- 172.21.15.5:0/2210993170 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f35d0125430 con 0x7f35d009f3d0 2024-09-16T10:59:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.949+0000 7f35c5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T10:59:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.949+0000 7f35c5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:55.958 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.958 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=10,kv=10,logm=65} crc {auth=4127400013,config=692619927,health=2345169553,kv=564032378,logm=287422713}) 2024-09-16T10:59:55.958 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=277155301,config=692619927,health=3623406937,kv=564032378,logm=2673302405}) 2024-09-16T10:59:55.958 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.958 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=10,kv=10,logm=65} crc {auth=4127400013,config=692619927,health=2345169553,kv=564032378,logm=287422713}) 2024-09-16T10:59:55.959 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.949+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=1,health=11,kv=10,logm=63} crc {auth=109035961,config=3678701417,health=305190204,kv=564032378,logm=3540443947}) 2024-09-16T10:59:55.961 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.962 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635776107}) 2024-09-16T10:59:55.962 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549796674}) 2024-09-16T10:59:55.962 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.962 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635776107}) 2024-09-16T10:59:55.962 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.953+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4132388218}) 2024-09-16T10:59:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 10 librados: watch_flush enter 2024-09-16T10:59:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 10 librados: watch_flush exit 2024-09-16T10:59:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 20 client.74130.objecter shutdown clearing up homeless session... 2024-09-16T10:59:55.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 10 client.74130.objecter successfully canceled tick 2024-09-16T10:59:55.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 msgr2=0x7f35b4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 0x7f35b4039420 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005150 tx=0x7f35c0007ee0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 10 monclient: shutdown 2024-09-16T10:59:55.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:55.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 msgr2=0x7f35d0138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35cc015040 tx=0x7f35cc004af0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 shutdown_connections 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f35d013f710 0x7f35d006fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> v2:172.21.15.143:6812/1098205034 conn(0x7f35b4036f60 0x7f35b4039420 unknown :-1 s=CLOSED pgs=3824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35d009f3d0 0x7f35d0138e60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 --2- 172.21.15.5:0/2210993170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35d009e8e0 0x7f35d0138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 >> 172.21.15.5:0/2210993170 conn(0x7f35d00a45d0 msgr2=0x7f35d00a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 shutdown_connections 2024-09-16T10:59:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 -- 172.21.15.5:0/2210993170 wait complete. 2024-09-16T10:59:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:55.957+0000 7f35d65f1700 1 librados: shutdown 2024-09-16T10:59:55.972 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.973 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179983980}) 2024-09-16T10:59:55.973 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355066322}) 2024-09-16T10:59:55.973 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.973 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179983980}) 2024-09-16T10:59:55.973 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.957+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589227074}) 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4035846640}) 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090618144}) 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4035846640}) 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.961+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780953950}) 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.974 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708416968}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062922131}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708416968}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31497079}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053874481}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974776337}) 2024-09-16T10:59:55.975 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.976 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053874481}) 2024-09-16T10:59:55.976 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749408406}) 2024-09-16T10:59:55.977 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.977 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684308066}) 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054758153}) 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684308066}) 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.965+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958198526}) 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658499810}) 2024-09-16T10:59:55.978 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736710609}) 2024-09-16T10:59:55.979 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.979 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658499810}) 2024-09-16T10:59:55.979 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542237147}) 2024-09-16T10:59:55.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T10:59:55.981 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770719691}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049558755}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770719691}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.969+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400723625}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248124924}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522653544}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248124924}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996168427}) 2024-09-16T10:59:55.982 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937910641}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1622760440}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937910641}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054559645}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389737084}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3704146409}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389737084}) 2024-09-16T10:59:55.983 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.973+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030918216}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089044650}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240465606}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089044650}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262785172}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986028609}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338211689}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986028609}) 2024-09-16T10:59:55.985 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494920758}) 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1799479446}) 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270492024}) 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1799479446}) 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.977+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320602945}) 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.987 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123225987}) 2024-09-16T10:59:55.988 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263245304}) 2024-09-16T10:59:55.988 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.988 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123225987}) 2024-09-16T10:59:55.988 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2592621874}) 2024-09-16T10:59:55.988 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.989 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248050998}) 2024-09-16T10:59:55.989 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290490202}) 2024-09-16T10:59:55.989 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.989 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248050998}) 2024-09-16T10:59:55.989 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122338874}) 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355161849}) 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107121797}) 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355161849}) 2024-09-16T10:59:55.990 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.981+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234510892}) 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264554877}) 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61766359}) 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264554877}) 2024-09-16T10:59:55.991 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239263812}) 2024-09-16T10:59:55.992 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.992 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824111632}) 2024-09-16T10:59:55.992 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341134239}) 2024-09-16T10:59:55.993 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.993 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824111632}) 2024-09-16T10:59:55.993 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2979928892}) 2024-09-16T10:59:55.994 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.994 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612579530}) 2024-09-16T10:59:55.994 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872171343}) 2024-09-16T10:59:55.994 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612579530}) 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.985+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1630810321}) 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022191055}) 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149530576}) 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022191055}) 2024-09-16T10:59:55.995 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232727774}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794450645}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004083958}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794450645}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335810156}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321343319}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387134}) 2024-09-16T10:59:55.998 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321343319}) 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2498645907}) 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644459287}) 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.989+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513348292}) 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644459287}) 2024-09-16T10:59:55.999 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364374235}) 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798599750}) 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393221596}) 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798599750}) 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751015267}) 2024-09-16T10:59:56.001 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235456430}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49813191}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235456430}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523482759}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740566835}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252579234}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740566835}) 2024-09-16T10:59:56.002 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.993+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813475066}) 2024-09-16T10:59:56.004 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.004 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707747859}) 2024-09-16T10:59:56.004 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310244297}) 2024-09-16T10:59:56.004 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707747859}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484981180}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040046336}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687821758}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040046336}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577212405}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2584375466}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449520271}) 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.005 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2584375466}) 2024-09-16T10:59:56.006 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1672968557}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913064304}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3406990121}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:55.997+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913064304}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=664357867}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036088592}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368395728}) 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.007 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036088592}) 2024-09-16T10:59:56.008 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272938515}) 2024-09-16T10:59:56.008 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.008 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040311951}) 2024-09-16T10:59:56.008 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771079865}) 2024-09-16T10:59:56.009 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.009 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040311951}) 2024-09-16T10:59:56.009 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196180425}) 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269124581}) 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520099319}) 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269124581}) 2024-09-16T10:59:56.010 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.001+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3892497732}) 2024-09-16T10:59:56.011 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425821187}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396747068}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425821187}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11909658}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187974845}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505366815}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1187974845}) 2024-09-16T10:59:56.012 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3653632537}) 2024-09-16T10:59:56.013 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283694103}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2292390084}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283694103}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.005+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298965742}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789949799}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596869920}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789949799}) 2024-09-16T10:59:56.014 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399956634}) 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2539586012,mdsmap=1531151339,mgr=477766456}) 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=2785436893,mdsmap=1531151339,mgr=1836277893}) 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2539586012,mdsmap=1531151339,mgr=477766456}) 2024-09-16T10:59:56.016 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.009+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=2,mgr=6} crc {logm=333509257,mdsmap=3585761405,mgr=477766456}) 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=25} crc {mgr=2586702517,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=348059101,mon_config_key=850781178,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=2792862242,osd_snap=1263314552,osdmap=700944639}) 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=26} crc {mgr=2586702516,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=2752295883,mon_config_key=2456723759,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=2792862242,osd_snap=2812485988,osdmap=136614740}) 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=25} crc {mgr=2586702517,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=348059101,mon_config_key=850781178,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=2792862242,osd_snap=1263314552,osdmap=700944639}) 2024-09-16T10:59:56.018 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.013+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=40,osdmap=23} crc {mgr=2586702516,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=96707212,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3111584180,osd_pg_creating=2792862242,osd_snap=3006885096,osdmap=3835747692}) 2024-09-16T10:59:56.029 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.029 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=293013143}) 2024-09-16T10:59:56.029 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=3084488075}) 2024-09-16T10:59:56.029 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T10:59:56.030 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=293013143}) 2024-09-16T10:59:56.030 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T10:59:56.021+0000 7efdf0ede700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=4027764191}) 2024-09-16T10:59:56.980 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T10:59:56.980 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:59:57.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: get_monmap_and_config 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: monmap: 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:57.098942+0000 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:57.098942+0000 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:57.100 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:57.101 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:57.101 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:57.101 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 Processor -- start 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 -- start start 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: init 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f455409e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): picked mon.noname-b con 0x7f455409f3d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 0x7f4554141b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): picked mon.noname-c con 0x7f455413f710 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455409fb80 con 0x7f455409e8e0 2024-09-16T10:59:57.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455409c030 con 0x7f455413f710 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455409f7b0 con 0x7f455409f3d0 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f4558807700 10 monclient(hunting): authenticate will time out at 1455.675337s 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37938/0 (socket says 172.21.15.5:37938) 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 1 -- 172.21.15.5:0/2111773575 learned_addr learned my addr 172.21.15.5:0/2111773575 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): get_auth_request con 0x7f455409f3d0 auth_method 0 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455359e700 1 --2- 172.21.15.5:0/2111773575 >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 0x7f4554141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455359e700 10 monclient(hunting): get_auth_request con 0x7f455413f710 auth_method 0 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.093+0000 7f455359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 10 monclient(hunting): handle_auth_done global_id 54362 payload 274 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 -- 172.21.15.5:0/2111773575 >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 msgr2=0x7f4554141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 --2- 172.21.15.5:0/2111773575 >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 0x7f4554141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 -- 172.21.15.5:0/2111773575 >> v1:172.21.15.143:6789/0 conn(0x7f455409e8e0 legacy=0x7f455409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 monclient: found mon.noname-b 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 -- 172.21.15.5:0/2111773575 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455409b970 con 0x7f455409f3d0 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455359e700 1 --2- 172.21.15.5:0/2111773575 >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 0x7f4554141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f455259c700 1 --2- 172.21.15.5:0/2111773575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4544009910 tx=0x7f4544009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e83665f0f5fdda7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 1 -- 172.21.15.5:0/2111773575 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400ba60 con 0x7f455409f3d0 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: dump: 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.114 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 5 monclient: authenticate success, global_id 54362 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:57.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 1 -- 172.21.15.5:0/2111773575 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454400bbc0 con 0x7f455409f3d0 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 4 set_mon_vals no callback set 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 1 -- 172.21.15.5:0/2111773575 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544009e60 con 0x7f455409f3d0 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4551d9b700 10 monclient: dump: 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.116 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 10 monclient: shutdown 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 -- 172.21.15.5:0/2111773575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 msgr2=0x7f455413f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 --2- 172.21.15.5:0/2111773575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4544009910 tx=0x7f4544009c20 comp rx=0 tx=0).stop 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 -- 172.21.15.5:0/2111773575 shutdown_connections 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 --2- 172.21.15.5:0/2111773575 >> v2:172.21.15.143:3300/0 conn(0x7f455413f710 0x7f4554141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 --2- 172.21.15.5:0/2111773575 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409f3d0 0x7f455413f1d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 -- 172.21.15.5:0/2111773575 >> 172.21.15.5:0/2111773575 conn(0x7f45540a45d0 msgr2=0x7f45540a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 -- 172.21.15.5:0/2111773575 shutdown_connections 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 1 -- 172.21.15.5:0/2111773575 wait complete. 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.097+0000 7f4558807700 10 monclient: monmap: 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:57.118 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 librados: starting msgr at 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 librados: starting objecter 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 Processor -- start 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- start start 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 librados: setting wanted keys 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 librados: calling monclient init 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: init 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:59:57.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 0x7f4554138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): picked mon.noname-c con 0x7f455409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): picked mon.noname-a con 0x7f455409f3d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f455413f710 0x7f455406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): picked mon.noname-b con 0x7f455413f710 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554071100 con 0x7f455409f3d0 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554124d00 con 0x7f455413f710 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455413f400 con 0x7f455409e8e0 2024-09-16T10:59:57.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient(hunting): authenticate will time out at 1455.681537s 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 0x7f4554138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35554/0 (socket says 172.21.15.5:35554) 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 0x7f4554138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37948/0 (socket says 172.21.15.5:37948) 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 -- 172.21.15.5:0/3279714055 learned_addr learned my addr 172.21.15.5:0/3279714055 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): get_auth_request con 0x7f455409f3d0 auth_method 0 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): get_auth_request con 0x7f455409e8e0 auth_method 0 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f455413f710 0x7f455406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): get_auth_request con 0x7f455413f710 auth_method 0 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4552d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient(hunting): handle_auth_done global_id 74133 payload 824 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 -- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 msgr2=0x7f4554138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 0x7f4554138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 -- 172.21.15.5:0/3279714055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f455413f710 msgr2=0x7f455406f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f455413f710 0x7f455406f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 -- 172.21.15.5:0/3279714055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544009570 con 0x7f455409f3d0 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455259c700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4544000c00 tx=0x7f4544004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 client.?.objecter ms_handle_connect 0x7f455409f3d0 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454401a650 con 0x7f455409f3d0 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: dump: 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.124 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454401f070 con 0x7f455409f3d0 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454401a9d0 con 0x7f455409f3d0 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f453b7fe700 10 monclient: dump: 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 5 monclient: authenticate success, global_id 74133 2024-09-16T10:59:57.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _renew_subs 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f455159a700 4 set_mon_vals no callback set 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455409b0d0 con 0x7f455409f3d0 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 client.74133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _renew_subs 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45541232f0 con 0x7f455409f3d0 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.101+0000 7f4558807700 1 librados: init done 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4544021030 con 0x7f455409f3d0 2024-09-16T10:59:57.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4558807700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4558807700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4558807700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455409ecc0 con 0x7f455409f3d0 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 1 --2- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 0x7f4540039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(400..400 src has 344..400) v4 ==== 5384+0+0 (secure 0 0 0) 0x7f4544048fe0 con 0x7f455409f3d0 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 10 client.74133.objecter ms_dispatch 0x7f4554000fd0 osd_map(400..400 src has 344..400) v4 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 3 client.74133.objecter handle_osd_map got epochs [400,400] > 0 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 3 client.74133.objecter handle_osd_map decoding full epoch 400 2024-09-16T10:59:57.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 20 client.74133.objecter dump_active .. 0 homeless 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4552d9d700 1 --2- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 0x7f4540039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4552d9d700 10 monclient: get_auth_request con 0x7f4540036f90 auth_method 0 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f453b7fe700 10 client.74133.objecter ms_handle_connect 0x7f4540036f90 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.105+0000 7f4552d9d700 1 --2- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 0x7f4540039450 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f4548000f60 tx=0x7f4548009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.117+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4544010380 con 0x7f455409f3d0 2024-09-16T10:59:57.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.117+0000 7f453b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.117+0000 7f453b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:57.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f4558807700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:59:57.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f4558807700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:57.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f4558807700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f45540721d0 con 0x7f455409f3d0 2024-09-16T10:59:57.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f453b7fe700 1 -- 172.21.15.5:0/3279714055 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f4544015020 con 0x7f455409f3d0 2024-09-16T10:59:57.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f453b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:57.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.353+0000 7f453b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:57.360 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:59:57.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 10 librados: watch_flush enter 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 10 librados: watch_flush exit 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 20 client.74133.objecter shutdown clearing up homeless session... 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 10 client.74133.objecter successfully canceled tick 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 msgr2=0x7f4540039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 0x7f4540039450 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f4548000f60 tx=0x7f4548009040 comp rx=0 tx=0).stop 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 10 monclient: shutdown 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 msgr2=0x7f4554138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4544000c00 tx=0x7f4544004770 comp rx=0 tx=0).stop 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 shutdown_connections 2024-09-16T10:59:57.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f455413f710 0x7f455406f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4540036f90 0x7f4540039450 unknown :-1 s=CLOSED pgs=3825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f455409f3d0 0x7f4554138890 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 --2- 172.21.15.5:0/3279714055 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f455409e8e0 0x7f4554138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 >> 172.21.15.5:0/3279714055 conn(0x7f45540a45d0 msgr2=0x7f45540a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 shutdown_connections 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 -- 172.21.15.5:0/3279714055 wait complete. 2024-09-16T10:59:57.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.357+0000 7f4558807700 1 librados: shutdown 2024-09-16T10:59:57.375 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T10:59:57.375 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T10:59:57.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T10:59:57.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: get_monmap_and_config 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: monmap: 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.486 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:57.485336+0000 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:57.485336+0000 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:57.487 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 Processor -- start 2024-09-16T10:59:57.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- start start 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: init 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0e40254f0 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 0x7ff0e40bd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-c con 0x7ff0e40233a0 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0e40b8950 0x7ff0e40bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-a con 0x7ff0e40b8950 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e401dfc0 con 0x7ff0e40254f0 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e401dd50 con 0x7ff0e40b8950 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e40b9340 con 0x7ff0e40233a0 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient(hunting): authenticate will time out at 1456.062135s 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0e40b8950 0x7ff0e40bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0e40b8950 0x7ff0e40bfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35558/0 (socket says 172.21.15.5:35558) 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4128449908 learned_addr learned my addr 172.21.15.5:0/4128449908 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): get_auth_request con 0x7ff0e40b8950 auth_method 0 2024-09-16T10:59:57.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 1 --2- 172.21.15.5:0/4128449908 >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 0x7ff0e40bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 10 monclient(hunting): get_auth_request con 0x7ff0e40233a0 auth_method 0 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_done global_id 74136 payload 274 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4128449908 >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 msgr2=0x7ff0e40bd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:57.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 --2- 172.21.15.5:0/4128449908 >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 0x7ff0e40bd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4128449908 >> v1:172.21.15.143:6789/0 conn(0x7ff0e40254f0 legacy=0x7ff0e401fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4128449908 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e401e2a0 con 0x7ff0e40b8950 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0ebfff700 1 --2- 172.21.15.5:0/4128449908 >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 0x7ff0e40bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f14c0700 1 --2- 172.21.15.5:0/4128449908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40b8950 0x7ff0e40bfcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec064060 tx=0x7ff0ec061e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b4154902f4d5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 1 -- 172.21.15.5:0/4128449908 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec090070 con 0x7ff0e40b8950 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: dump: 2024-09-16T10:59:57.501 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.502 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 5 monclient: authenticate success, global_id 74136 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 1 -- 172.21.15.5:0/4128449908 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ec066e00 con 0x7ff0e40b8950 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 4 set_mon_vals no callback set 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 1 -- 172.21.15.5:0/4128449908 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec0008d0 con 0x7ff0e40b8950 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0eb7fe700 10 monclient: dump: 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 10 monclient: shutdown 2024-09-16T10:59:57.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4128449908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40b8950 msgr2=0x7ff0e40bfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4128449908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40b8950 0x7ff0e40bfcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec064060 tx=0x7ff0ec061e90 comp rx=0 tx=0).stop 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4128449908 shutdown_connections 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4128449908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40b8950 0x7ff0e40bfcb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4128449908 >> v2:172.21.15.143:3300/0 conn(0x7ff0e40233a0 0x7ff0e40bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.481+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4128449908 >> 172.21.15.5:0/4128449908 conn(0x7ff0e40064b0 msgr2=0x7ff0e40068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4128449908 shutdown_connections 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4128449908 wait complete. 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: monmap: 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.505 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 librados: starting msgr at 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 librados: starting objecter 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 Processor -- start 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- start start 2024-09-16T10:59:57.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 librados: setting wanted keys 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 librados: calling monclient init 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: init 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e40233a0 0x7ff0e40b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-c con 0x7ff0e40233a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:57.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 0x7ff0e40b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-a con 0x7ff0e40254f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e40b8950 0x7ff0e40a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): picked mon.noname-b con 0x7ff0e40b8950 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40a99e0 con 0x7ff0e40254f0 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4151670 con 0x7ff0e40b8950 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4020870 con 0x7ff0e40233a0 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient(hunting): authenticate will time out at 1456.066336s 2024-09-16T10:59:57.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 0x7ff0e40b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e40233a0 0x7ff0e40b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e40233a0 0x7ff0e40b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37956/0 (socket says 172.21.15.5:37956) 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 0x7ff0e40b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35572/0 (socket says 172.21.15.5:35572) 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 1 -- 172.21.15.5:0/4051179228 learned_addr learned my addr 172.21.15.5:0/4051179228 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): get_auth_request con 0x7ff0e40233a0 auth_method 0 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): get_auth_request con 0x7ff0e40254f0 auth_method 0 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e40b8950 0x7ff0e40a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): get_auth_request con 0x7ff0e40b8950 auth_method 0 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f14c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient(hunting): handle_auth_done global_id 74139 payload 824 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e40233a0 msgr2=0x7ff0e40b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff0e40233a0 0x7ff0e40b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e40b8950 msgr2=0x7ff0e40a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e40b8950 0x7ff0e40a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec069b60 con 0x7ff0e40254f0 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 client.?.objecter ms_handle_connect 0x7ff0e40254f0 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0ebfff700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 0x7ff0e40b59b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000e840 tx=0x7ff0e000eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e000c070 con 0x7ff0e40254f0 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: dump: 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.512 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 5 monclient: authenticate success, global_id 74139 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: _renew_subs 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0013610 con 0x7ff0e40254f0 2024-09-16T10:59:57.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e4028f00 con 0x7ff0e40254f0 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00139c0 con 0x7ff0e40254f0 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0f1cc1700 10 client.74139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0eaffd700 4 set_mon_vals no callback set 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.485+0000 7ff0e97fa700 10 monclient: dump: 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.514 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 10 monclient: _renew_subs 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e40220f0 con 0x7ff0e40254f0 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 1 librados: init done 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 0x7ff0e40a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e4007f80 con 0x7ff0e4150930 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 0x7ff0e40a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient: get_auth_request con 0x7ff0e4150930 auth_method 0 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_done global_id 74142 payload 824 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e4020b80 con 0x7ff0e4150930 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff0e0013b20 con 0x7ff0e40254f0 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f0cbf700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 0x7ff0e40a64d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800f6c0 tx=0x7ff0d800fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 --2- 172.21.15.5:0/4051179228 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0d0036f90 0x7ff0d0039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 client.74139.objecter ms_handle_connect 0x7ff0e4150930 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 client.74139.objecter resend_mon_ops 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(400..400 src has 344..400) v4 ==== 5384+0+0 (secure 0 0 0) 0x7ff0e0047a80 con 0x7ff0e40254f0 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 client.74139.objecter ms_dispatch 0x7ff0ec000fd0 osd_map(400..400 src has 344..400) v4 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 3 client.74139.objecter handle_osd_map got epochs [400,400] > 0 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 3 client.74139.objecter handle_osd_map decoding full epoch 400 2024-09-16T10:59:57.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 20 client.74139.objecter dump_active .. 0 homeless 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80052e0 con 0x7ff0e4150930 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 monclient: dump: 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.518 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f14c0700 1 --2- 172.21.15.5:0/4051179228 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0d0036f90 0x7ff0d0039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f14c0700 10 monclient: get_auth_request con 0x7ff0d0036f90 auth_method 0 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff0d8005440 con 0x7ff0e4150930 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 msgr2=0x7ff0e40a64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 0x7ff0e40a64d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800f6c0 tx=0x7ff0d800fa80 comp rx=0 tx=0).stop 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 msgr2=0x7ff0e40a64d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e4150930 0x7ff0e40a64d0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800f6c0 tx=0x7ff0d800fa80 comp rx=0 tx=0).stop 2024-09-16T10:59:57.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0e97fa700 10 client.74139.objecter ms_handle_connect 0x7ff0d0036f90 2024-09-16T10:59:57.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.489+0000 7ff0f14c0700 1 --2- 172.21.15.5:0/4051179228 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0d0036f90 0x7ff0d0039450 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009d30 tx=0x7ff0ec096040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f1cc1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:57.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 0x7ff0e4002970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f1cc1700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e40a6e20 con 0x7ff0e40a6a10 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 0x7ff0e4002970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient: get_auth_request con 0x7ff0e40a6a10 auth_method 0 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 10 monclient(hunting): handle_auth_done global_id 74145 payload 824 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 1 -- 172.21.15.5:0/4051179228 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff0e40a9fc0 con 0x7ff0e40a6a10 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 client.74139.objecter ms_handle_connect 0x7ff0e40a6a10 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 client.74139.objecter resend_mon_ops 2024-09-16T10:59:57.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0f0cbf700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 0x7ff0e4002970 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8017000 tx=0x7ff0d8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8023070 con 0x7ff0e40a6a10 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 monclient: dump: 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.524 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7ff0d8003c40 con 0x7ff0e40a6a10 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 msgr2=0x7ff0e4002970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 0x7ff0e4002970 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8017000 tx=0x7ff0d8001120 comp rx=0 tx=0).stop 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 msgr2=0x7ff0e4002970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.501+0000 7ff0e97fa700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40a6a10 0x7ff0e4002970 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8017000 tx=0x7ff0d8001120 comp rx=0 tx=0).stop 2024-09-16T10:59:57.525 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:57.526 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:57.527 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:57.528 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:57.529 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:57.530 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.531 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.532 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.533 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.534 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.535 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.536 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:57.537 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:57.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 10 librados: watch_flush enter 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 10 librados: watch_flush exit 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 20 client.74139.objecter shutdown clearing up homeless session... 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 10 client.74139.objecter successfully canceled tick 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0d0036f90 msgr2=0x7ff0d0039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4051179228 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff0d0036f90 0x7ff0d0039450 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009d30 tx=0x7ff0ec096040 comp rx=0 tx=0).stop 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 10 monclient: shutdown 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 msgr2=0x7ff0e40b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 --2- 172.21.15.5:0/4051179228 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0e40254f0 0x7ff0e40b59b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000e840 tx=0x7ff0e000eb50 comp rx=0 tx=0).stop 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4051179228 reap_dead start 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f14c0700 1 -- 172.21.15.5:0/4051179228 reap_dead start 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 shutdown_connections 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 >> 172.21.15.5:0/4051179228 conn(0x7ff0e40064b0 msgr2=0x7ff0e4020f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 shutdown_connections 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 -- 172.21.15.5:0/4051179228 wait complete. 2024-09-16T10:59:57.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.505+0000 7ff0f1cc1700 1 librados: shutdown 2024-09-16T10:59:57.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:57.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T10:59:57.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T10:59:57.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: get_monmap_and_config 2024-09-16T10:59:57.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: monmap: 2024-09-16T10:59:57.662 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.662 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:57.661511+0000 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:57.661511+0000 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:57.663 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 Processor -- start 2024-09-16T10:59:57.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- start start 2024-09-16T10:59:57.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: init 2024-09-16T10:59:57.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f880a1ad0 0x7f0f880a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-c con 0x7f0f880a1ad0 addr v2:172.21.15.143:3300/0 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0f880a1eb0 0x7f0f880a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-a con 0x7f0f880a1eb0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f88139c90 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8809de20 con 0x7f0f88139c90 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8809fc70 con 0x7f0f880a1eb0 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8809e020 con 0x7f0f880a1ad0 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient(hunting): authenticate will time out at 1456.238121s 2024-09-16T10:59:57.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0f880a1eb0 0x7f0f880a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0f880a1eb0 0x7f0f880a22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35586/0 (socket says 172.21.15.5:35586) 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/1987537061 learned_addr learned my addr 172.21.15.5:0/1987537061 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): get_auth_request con 0x7f0f880a1eb0 auth_method 0 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f87fff700 1 --2- 172.21.15.5:0/1987537061 >> v2:172.21.15.143:3300/0 conn(0x7f0f880a1ad0 0x7f0f880a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f87fff700 10 monclient(hunting): get_auth_request con 0x7f0f880a1ad0 auth_method 0 2024-09-16T10:59:57.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_done global_id 74148 payload 274 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/1987537061 >> v2:172.21.15.143:3300/0 conn(0x7f0f880a1ad0 msgr2=0x7f0f880a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 --2- 172.21.15.5:0/1987537061 >> v2:172.21.15.143:3300/0 conn(0x7f0f880a1ad0 0x7f0f880a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/1987537061 >> v1:172.21.15.143:6789/0 conn(0x7f0f88139c90 legacy=0x7f0f8813c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/1987537061 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8809f340 con 0x7f0f880a1eb0 2024-09-16T10:59:57.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f7f7fe700 1 --2- 172.21.15.5:0/1987537061 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1eb0 0x7f0f880a22e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f880a3ae0 tx=0x7f0f70009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637250833e3ffd1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 1 -- 172.21.15.5:0/1987537061 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70004230 con 0x7f0f880a1eb0 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: dump: 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.675 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 5 monclient: authenticate success, global_id 74148 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 1 -- 172.21.15.5:0/1987537061 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70004390 con 0x7f0f880a1eb0 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 4 set_mon_vals no callback set 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 1 -- 172.21.15.5:0/1987537061 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700036c0 con 0x7f0f880a1eb0 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f877fe700 10 monclient: dump: 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 10 monclient: shutdown 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/1987537061 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1eb0 msgr2=0x7f0f880a22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/1987537061 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1eb0 0x7f0f880a22e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f880a3ae0 tx=0x7f0f70009bf0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/1987537061 shutdown_connections 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/1987537061 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1eb0 0x7f0f880a22e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.657+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/1987537061 >> v2:172.21.15.143:3300/0 conn(0x7f0f880a1ad0 0x7f0f880a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/1987537061 >> 172.21.15.5:0/1987537061 conn(0x7f0f880a68a0 msgr2=0x7f0f880a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/1987537061 shutdown_connections 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/1987537061 wait complete. 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: monmap: 2024-09-16T10:59:57.678 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 librados: starting msgr at 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 librados: starting objecter 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 Processor -- start 2024-09-16T10:59:57.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- start start 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 librados: setting wanted keys 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 librados: calling monclient init 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: init 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1ad0 0x7f0f881c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-a con 0x7f0f880a1ad0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-c con 0x7f0f880a1eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f88139c90 0x7f0f881ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): picked mon.noname-b con 0x7f0f88139c90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:57.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88125580 con 0x7f0f880a1ad0 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88099a40 con 0x7f0f88139c90 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88123320 con 0x7f0f880a1eb0 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient(hunting): authenticate will time out at 1456.241964s 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1ad0 0x7f0f881c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37958/0 (socket says 172.21.15.5:37958) 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/181802630 learned_addr learned my addr 172.21.15.5:0/181802630 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): get_auth_request con 0x7f0f880a1eb0 auth_method 0 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): get_auth_request con 0x7f0f880a1ad0 auth_method 0 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f88139c90 0x7f0f881ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): get_auth_request con 0x7f0f88139c90 auth_method 0 2024-09-16T10:59:57.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8cbab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient(hunting): handle_auth_done global_id 54371 payload 824 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f88139c90 msgr2=0x7f0f881ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f88139c90 0x7f0f881ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1ad0 msgr2=0x7f0f881c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1ad0 0x7f0f881c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 monclient: found mon.noname-c 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009570 con 0x7f0f880a1eb0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f87fff700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f880a1ad0 0x7f0f881c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f7f7fe700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0f70004000 tx=0x7f0f70004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 client.?.objecter ms_handle_connect 0x7f0f880a1eb0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f700036e0 con 0x7f0f880a1eb0 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T10:59:57.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: dump: 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 5 monclient: authenticate success, global_id 54371 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _renew_subs 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7000b660 con 0x7f0f880a1eb0 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8809d2b0 con 0x7f0f880a1eb0 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000ba10 con 0x7f0f880a1eb0 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 client.54371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f86ffd700 4 set_mon_vals no callback set 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f857fa700 10 monclient: dump: 2024-09-16T10:59:57.686 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _renew_subs 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f88125e20 con 0x7f0f880a1eb0 2024-09-16T10:59:57.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.661+0000 7f0f8d3ac700 1 librados: init done 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0f70021070 con 0x7f0f880a1eb0 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8d3ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 0x7f0f881ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88067380 con 0x7f0f881389a0 2024-09-16T10:59:57.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 --2- 172.21.15.5:0/181802630 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0f68036f60 0x7f0f68039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 0x7f0f881ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(401..401 src has 344..401) v4 ==== 5471+0+0 (secure 0 0 0) 0x7f0f700495c0 con 0x7f0f880a1eb0 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 client.54371.objecter ms_dispatch 0x7f0f88000fd0 osd_map(401..401 src has 344..401) v4 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 3 client.54371.objecter handle_osd_map got epochs [401,401] > 0 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient: get_auth_request con 0x7f0f881389a0 auth_method 0 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 3 client.54371.objecter handle_osd_map decoding full epoch 401 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 20 client.54371.objecter dump_active .. 0 homeless 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 10 monclient(hunting): handle_auth_done global_id 54374 payload 824 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f8809f6e0 con 0x7f0f881389a0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8cbab700 1 --2- 172.21.15.5:0/181802630 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0f68036f60 0x7f0f68039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8cbab700 10 monclient: get_auth_request con 0x7f0f68036f60 auth_method 0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 client.54371.objecter ms_handle_connect 0x7f0f881389a0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f87fff700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 0x7f0f881ca8f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f78011a80 tx=0x7f0f78011e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 client.54371.objecter resend_mon_ops 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800c070 con 0x7f0f881389a0 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:57.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 monclient: dump: 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f0f7800fb20 con 0x7f0f881389a0 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:57.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 msgr2=0x7f0f881ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 0x7f0f881ca8f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f78011a80 tx=0x7f0f78011e40 comp rx=0 tx=0).stop 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 msgr2=0x7f0f881ca8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881389a0 0x7f0f881ca8f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f857fa700 10 client.54371.objecter ms_handle_connect 0x7f0f68036f60 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.665+0000 7f0f8cbab700 1 --2- 172.21.15.5:0/181802630 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0f68036f60 0x7f0f68039420 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000f2e0 tx=0x7f0f80004260 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 0x7f0f8806e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8809e510 con 0x7f0f881cae30 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 0x7f0f8806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient: get_auth_request con 0x7f0f881cae30 auth_method 0 2024-09-16T10:59:57.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 10 monclient(hunting): handle_auth_done global_id 54377 payload 824 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 1 -- 172.21.15.5:0/181802630 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0f881cb240 con 0x7f0f881cae30 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f87fff700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 0x7f0f8806e330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009b40 tx=0x7f0f78004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 client.54371.objecter ms_handle_connect 0x7f0f881cae30 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 client.54371.objecter resend_mon_ops 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78024070 con 0x7f0f881cae30 2024-09-16T10:59:57.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 monclient: dump: 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f0f78003e50 con 0x7f0f881cae30 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 msgr2=0x7f0f8806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 0x7f0f8806e330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009b40 tx=0x7f0f78004770 comp rx=0 tx=0).stop 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 msgr2=0x7f0f8806e330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f857fa700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f881cae30 0x7f0f8806e330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:57.698 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.699 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.700 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.701 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:57.702 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.703 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.704 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.705 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.706 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:57.707 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.708 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.709 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.710 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:59:57.711 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.711 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.711 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.711 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:57.711 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 librados: watch_flush enter 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 librados: watch_flush exit 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 20 client.54371.objecter shutdown clearing up homeless session... 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 client.54371.objecter successfully canceled tick 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0f68036f60 msgr2=0x7f0f68039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/181802630 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0f68036f60 0x7f0f68039420 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000f2e0 tx=0x7f0f80004260 comp rx=0 tx=0).stop 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 10 monclient: shutdown 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 msgr2=0x7f0f88138460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8cbab700 1 -- 172.21.15.5:0/181802630 reap_dead start 2024-09-16T10:59:57.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0f70004000 tx=0x7f0f70004770 comp rx=0 tx=0).stop 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 shutdown_connections 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 --2- 172.21.15.5:0/181802630 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0f880a1eb0 0x7f0f88138460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 >> 172.21.15.5:0/181802630 conn(0x7f0f880a68a0 msgr2=0x7f0f880a4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 shutdown_connections 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 -- 172.21.15.5:0/181802630 wait complete. 2024-09-16T10:59:57.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.677+0000 7f0f8d3ac700 1 librados: shutdown 2024-09-16T10:59:57.714 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:57.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T10:59:57.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T10:59:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: get_monmap_and_config 2024-09-16T10:59:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: monmap: 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:57.833787+0000 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:57.833787+0000 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:57.835 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:57.836 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:57.836 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:57.836 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 Processor -- start 2024-09-16T10:59:57.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 -- start start 2024-09-16T10:59:57.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: init 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa6c09e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa6c09d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa6c09dfd0 0x7ffa6c13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-a con 0x7ffa6c09dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c070e40 con 0x7ffa6c09d6a0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c09fef0 con 0x7ffa6c09e8e0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c0a01d0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient(hunting): authenticate will time out at 1456.410440s 2024-09-16T10:59:57.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa6c09dfd0 0x7ffa6c13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa70d59700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ffa6c09d6a0 0x7ffa6c138b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46612/0 (socket says 172.21.15.5:46612) 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa70d59700 1 -- 172.21.15.5:0/1588776013 learned_addr learned my addr 172.21.15.5:0/1588776013 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): get_auth_request con 0x7ffa6c09dfd0 auth_method 0 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient(hunting): handle_auth_done global_id 74154 payload 274 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 -- 172.21.15.5:0/1588776013 >> v1:172.21.15.143:6789/0 conn(0x7ffa6c09e8e0 legacy=0x7ffa6c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 -- 172.21.15.5:0/1588776013 >> v1:172.21.15.5:6789/0 conn(0x7ffa6c09d6a0 legacy=0x7ffa6c138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 -- 172.21.15.5:0/1588776013 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c09f9f0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa71d5b700 1 --2- 172.21.15.5:0/1588776013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c13c2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009880 tx=0x7ffa64009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc00d16ec373dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 1 -- 172.21.15.5:0/1588776013 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64011830 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: dump: 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.848 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 5 monclient: authenticate success, global_id 74154 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 1 -- 172.21.15.5:0/1588776013 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64011990 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 1 -- 172.21.15.5:0/1588776013 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64011d20 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa7255c700 4 set_mon_vals no callback set 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.829+0000 7ffa63fff700 10 monclient: dump: 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: shutdown 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1588776013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 msgr2=0x7ffa6c13c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1588776013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c13c2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009880 tx=0x7ffa64009b90 comp rx=0 tx=0).stop 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1588776013 shutdown_connections 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1588776013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c13c2b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1588776013 >> 172.21.15.5:0/1588776013 conn(0x7ffa6c0a45d0 msgr2=0x7ffa6c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1588776013 shutdown_connections 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1588776013 wait complete. 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: build_initial_monmap 2024-09-16T10:59:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: monmap: 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 librados: starting msgr at 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 librados: starting objecter 2024-09-16T10:59:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 Processor -- start 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- start start 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 librados: setting wanted keys 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 librados: calling monclient init 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: init 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa6c09d6a0 0x7ffa6c1cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-c con 0x7ffa6c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-a con 0x7ffa6c09dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c09e8e0 0x7ffa6c122340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): picked mon.noname-b con 0x7ffa6c09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c126280 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c134700 con 0x7ffa6c09e8e0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c06cf10 con 0x7ffa6c09d6a0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient(hunting): authenticate will time out at 1456.414332s 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa6c09d6a0 0x7ffa6c1cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35614/0 (socket says 172.21.15.5:35614) 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 -- 172.21.15.5:0/1476168787 learned_addr learned my addr 172.21.15.5:0/1476168787 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): get_auth_request con 0x7ffa6c09dfd0 auth_method 0 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): get_auth_request con 0x7ffa6c09d6a0 auth_method 0 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient(hunting): handle_auth_done global_id 74160 payload 824 2024-09-16T10:59:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa71d5b700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c09e8e0 0x7ffa6c122340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa6c09d6a0 msgr2=0x7ffa6c1cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa6c09d6a0 0x7ffa6c1cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c09e8e0 msgr2=0x7ffa6c122340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c09e8e0 0x7ffa6c122340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient: _finish_hunting 0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 monclient: found mon.noname-a 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa640094b0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa71d5b700 10 monclient: get_auth_request con 0x7ffa6c09e8e0 auth_method 0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa71d5b700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c09e8e0 0x7ffa6c122340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa70d59700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800cba0 tx=0x7ffa6800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 client.?.objecter ms_handle_connect 0x7ffa6c09dfd0 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800a7f0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: dump: 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: _finish_auth 0 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 5 monclient: authenticate success, global_id 74160 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _renew_subs 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6800a950 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c124a80 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800acc0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 client.74160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa62ffd700 4 set_mon_vals no callback set 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa61ffb700 10 monclient: dump: 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _renew_subs 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c1259e0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.833+0000 7ffa7255c700 1 librados: init done 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7255c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 0x7ffa6c122c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c067380 con 0x7ffa6c122880 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ffa6801b3b0 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 1 --2- 172.21.15.5:0/1476168787 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa50036f60 0x7ffa50039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 0x7ffa6c122c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(401..401 src has 344..401) v4 ==== 5471+0+0 (secure 0 0 0) 0x7ffa68047840 con 0x7ffa6c09dfd0 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 client.74160.objecter ms_dispatch 0x7ffa6c000fd0 osd_map(401..401 src has 344..401) v4 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 3 client.74160.objecter handle_osd_map got epochs [401,401] > 0 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 3 client.74160.objecter handle_osd_map decoding full epoch 401 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient: get_auth_request con 0x7ffa6c122880 auth_method 0 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 20 client.74160.objecter dump_active .. 0 homeless 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa71d5b700 1 --2- 172.21.15.5:0/1476168787 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa50036f60 0x7ffa50039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa71d5b700 10 monclient: get_auth_request con 0x7ffa50036f60 auth_method 0 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 10 monclient(hunting): handle_auth_done global_id 64264 payload 824 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa6c075900 con 0x7ffa6c122880 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa7155a700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 0x7ffa6c122c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c00fb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 client.74160.objecter ms_handle_connect 0x7ffa6c122880 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 client.74160.objecter resend_mon_ops 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 client.74160.objecter ms_handle_connect 0x7ffa50036f60 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa71d5b700 1 --2- 172.21.15.5:0/1476168787 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa50036f60 0x7ffa50039420 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c09e3d0 tx=0x7ffa64008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c005320 con 0x7ffa6c122880 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 10 monclient: dump: 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.837+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ffa5c005480 con 0x7ffa6c122880 2024-09-16T10:59:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 msgr2=0x7ffa6c122c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 0x7ffa6c122c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c00fb60 comp rx=0 tx=0).stop 2024-09-16T10:59:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 msgr2=0x7ffa6c122c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.841+0000 7ffa61ffb700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c122880 0x7ffa6c122c60 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c00fb60 comp rx=0 tx=0).stop 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.849+0000 7ffa7255c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.849+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 0x7ffa6c0a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.849+0000 7ffa7255c700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.849+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c09f3d0 con 0x7ffa6c1231a0 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 0x7ffa6c0a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient: get_auth_request con 0x7ffa6c1231a0 auth_method 0 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 10 monclient(hunting): handle_auth_done global_id 64267 payload 824 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 1 -- 172.21.15.5:0/1476168787 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa6c09ecc0 con 0x7ffa6c1231a0 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7155a700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 0x7ffa6c0a1fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 client.74160.objecter ms_handle_connect 0x7ffa6c1231a0 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 client.74160.objecter resend_mon_ops 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c023070 con 0x7ffa6c1231a0 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T10:59:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 monclient: dump: 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7ffa5c007640 con 0x7ffa6c1231a0 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T10:59:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 msgr2=0x7ffa6c0a1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 0x7ffa6c0a1fb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c007f90 comp rx=0 tx=0).stop 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 msgr2=0x7ffa6c0a1fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa61ffb700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa6c1231a0 0x7ffa6c0a1fb0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00f7a0 tx=0x7ffa5c007f90 comp rx=0 tx=0).stop 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 44, 2024-09-16T10:59:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T10:59:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T10:59:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T10:59:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T10:59:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T10:59:57.880 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T10:59:57.881 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.882 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T10:59:57.883 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T10:59:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T10:59:57.884 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T10:59:57.884 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T10:59:57.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 10 librados: watch_flush enter 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 10 librados: watch_flush exit 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 20 client.74160.objecter shutdown clearing up homeless session... 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 10 client.74160.objecter successfully canceled tick 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa50036f60 msgr2=0x7ffa50039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1476168787 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa50036f60 0x7ffa50039420 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c09e3d0 tx=0x7ffa64008030 comp rx=0 tx=0).stop 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 10 monclient: shutdown 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 msgr2=0x7ffa6c1cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800cba0 tx=0x7ffa6800cbd0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa71d5b700 1 -- 172.21.15.5:0/1476168787 reap_dead start 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 shutdown_connections 2024-09-16T10:59:57.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 --2- 172.21.15.5:0/1476168787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa6c09dfd0 0x7ffa6c1cb130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:57.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 >> 172.21.15.5:0/1476168787 conn(0x7ffa6c0a45d0 msgr2=0x7ffa6c133fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:57.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 shutdown_connections 2024-09-16T10:59:57.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 -- 172.21.15.5:0/1476168787 wait complete. 2024-09-16T10:59:57.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:57.853+0000 7ffa7255c700 1 librados: shutdown 2024-09-16T10:59:57.887 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T10:59:57.887 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T10:59:57.887 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T10:59:57.888 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T10:59:57.888 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T10:59:57.888 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T10:59:57.888 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:59:57.934 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T10:59:57.934 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T10:59:57.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.041+0000 7f9b2a02e700 10 monclient: get_monmap_and_config 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.041+0000 7f9b2a02e700 10 monclient: build_initial_monmap 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.041+0000 7f9b2a02e700 10 monclient: monmap: 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:59:58.048478+0000 2024-09-16T10:59:58.049 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:59:58.048478+0000 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T10:59:58.050 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 Processor -- start 2024-09-16T10:59:58.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 -- start start 2024-09-16T10:59:58.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient: init 2024-09-16T10:59:58.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:58.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b240a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b2413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-b con 0x7f9b240a3770 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b2413b1c0 addr v1:172.21.15.5:6789/0 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b240a3db0 con 0x7f9b2413b1c0 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b240a4b70 con 0x7f9b240a2c80 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240a3b50 con 0x7f9b240a3770 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 10 monclient(hunting): authenticate will time out at 1456.625251s 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2882b700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 msgr2=0x7f9b2413ac80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2882b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b2413ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2982d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9b2413b1c0 0x7f9b2413d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46622/0 (socket says 172.21.15.5:46622) 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2982d700 1 -- 172.21.15.5:0/3924420009 learned_addr learned my addr 172.21.15.5:0/3924420009 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 --1- 172.21.15.5:0/3924420009 >> v1:172.21.15.143:6789/0 conn(0x7f9b240a2c80 0x7f9b240a3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:41230/0 (socket says 172.21.15.5:41230) 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454392497 0 0) 0x7f9b240a4b70 con 0x7f9b240a2c80 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _negotiate 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): my global_id is 64270 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b100035d0 con 0x7f9b240a2c80 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049629952 0 0) 0x7f9b240a3db0 con 0x7f9b2413b1c0 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _negotiate 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): my global_id is 74163 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b240a4b70 con 0x7f9b2413b1c0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656921510 0 0) 0x7f9b240a4b70 con 0x7f9b2413b1c0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 msgr2=0x7f9b2413ac80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b2413ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 >> v1:172.21.15.143:6789/0 conn(0x7f9b240a2c80 legacy=0x7f9b240a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _finish_hunting 0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 monclient: found mon.noname-a-legacy 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240a0850 con 0x7f9b2413b1c0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _finish_auth 0 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _check_auth_tickets 2024-09-16T10:59:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 5 monclient: authenticate success, global_id 74163 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f9b1c001650 con 0x7f9b2413b1c0 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: dump: 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 >> v1:172.21.15.5:6789/0 conn(0x7f9b2413b1c0 legacy=0x7f9b2413d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b10004c70 0x7f9b10005050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): picked mon.a con 0x7f9b10004c70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b10003810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): picked mon.c con 0x7f9b240a3770 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b10003e70 0x7f9b1000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): picked mon.b con 0x7f9b10003e70 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b20001230 con 0x7f9b10004c70 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240a3b50 con 0x7f9b10003e70 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b100046e0 con 0x7f9b240a3770 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b17fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2a02e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2882b700 1 -- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 msgr2=0x7f9b10003810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b10004c70 0x7f9b10005050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2882b700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b10003810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): get_auth_request con 0x7f9b10004c70 auth_method 0 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient(hunting): handle_auth_done global_id 74163 payload 293 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 -- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 msgr2=0x7f9b10003810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b240a3770 0x7f9b10003810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 -- 172.21.15.5:0/3924420009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b10003e70 msgr2=0x7f9b1000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T10:59:58.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b10003e70 0x7f9b1000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient: _finish_hunting 0 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 monclient: found mon.a 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 -- 172.21.15.5:0/3924420009 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240a4b70 con 0x7f9b10004c70 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient: _finish_auth 0 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 10 monclient: _check_auth_tickets 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.045+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b10004c70 0x7f9b10005050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c110 tx=0x7f9b100058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1d115dca4a7eb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20007a70 con 0x7f9b10004c70 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: dump: 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2000cc30 con 0x7f9b10004c70 2024-09-16T10:59:58.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 10 monclient: get_monmap_and_config success 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 4 set_mon_vals no callback set 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 1 -- 172.21.15.5:0/3924420009 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20004890 con 0x7f9b10004c70 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b17fff700 10 monclient: dump: 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.067 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 10 monclient: shutdown 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b10004c70 msgr2=0x7f9b10005050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3924420009 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b10004c70 0x7f9b10005050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c110 tx=0x7f9b100058a0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2982d700 1 -- 172.21.15.5:0/3924420009 reap_dead start 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3924420009 shutdown_connections 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3924420009 >> 172.21.15.5:0/3924420009 conn(0x7f9b240a4d80 msgr2=0x7f9b240a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3924420009 shutdown_connections 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3924420009 wait complete. 2024-09-16T10:59:58.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 10 monclient: build_initial_monmap 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 10 monclient: monmap: 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 librados: starting msgr at 2024-09-16T10:59:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 librados: starting objecter 2024-09-16T10:59:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 Processor -- start 2024-09-16T10:59:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 -- start start 2024-09-16T10:59:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 librados: setting wanted keys 2024-09-16T10:59:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 1 librados: calling monclient init 2024-09-16T10:59:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.049+0000 7f9b2a02e700 10 monclient: init 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _reopen_session rank -1 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b241c7400 0x7f9b24136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-b con 0x7f9b241c7400 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-a con 0x7f9b241366d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 0x7f9b241cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): picked mon.noname-c con 0x7f9b241cae40 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24122580 con 0x7f9b241366d0 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24070540 con 0x7f9b241c7400 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): start opening mon connection 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b241c77e0 con 0x7f9b241cae40 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): _renew_subs 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient(hunting): authenticate will time out at 1456.632695s 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2982d700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 msgr2=0x7f9b241cb2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2982d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 0x7f9b241cb2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35636/0 (socket says 172.21.15.5:35636) 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 -- 172.21.15.5:0/3821512999 learned_addr learned my addr 172.21.15.5:0/3821512999 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T10:59:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): get_auth_request con 0x7f9b241366d0 auth_method 0 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b241c7400 0x7f9b24136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): get_auth_request con 0x7f9b241c7400 auth_method 0 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): _init_auth method 2 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T10:59:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2902c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient(hunting): handle_auth_done global_id 74166 payload 824 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 -- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 msgr2=0x7f9b241cb2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 0x7f9b241cb2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 -- 172.21.15.5:0/3821512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b241c7400 msgr2=0x7f9b24136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b241c7400 0x7f9b24136190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient: _finish_hunting 0 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 monclient: found mon.noname-a 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 -- 172.21.15.5:0/3821512999 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240a76b0 con 0x7f9b241366d0 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 client.?.objecter ms_handle_connect 0x7f9b241366d0 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2882b700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800b9e0 tx=0x7f9b1800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:58.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1800c950 con 0x7f9b241366d0 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: dump: 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: _finish_auth 0 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: _check_auth_tickets 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 5 monclient: authenticate success, global_id 74166 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1800cad0 con 0x7f9b241366d0 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _renew_subs 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b241222a0 con 0x7f9b241366d0 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1800ce80 con 0x7f9b241366d0 2024-09-16T10:59:58.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 monclient: dump: 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T10:59:58.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 client.74166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b16ffd700 4 set_mon_vals no callback set 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _renew_subs 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b24120200 con 0x7f9b241366d0 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 librados: init done 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9b18012070 con 0x7f9b241366d0 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:58.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b241227a0 con 0x7f9b241366d0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 --2- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 0x7f9b080393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(401..401 src has 344..401) v4 ==== 5471+0+0 (secure 0 0 0) 0x7f9b18047aa0 con 0x7f9b241366d0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 10 client.74166.objecter ms_dispatch 0x7f9b24000fd0 osd_map(401..401 src has 344..401) v4 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 3 client.74166.objecter handle_osd_map got epochs [401,401] > 0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.053+0000 7f9b15ffb700 3 client.74166.objecter handle_osd_map decoding full epoch 401 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.057+0000 7f9b15ffb700 20 client.74166.objecter dump_active .. 0 homeless 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.057+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 0x7f9b080393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.057+0000 7f9b2902c700 10 monclient: get_auth_request con 0x7f9b08036f10 auth_method 0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.057+0000 7f9b15ffb700 10 client.74166.objecter ms_handle_connect 0x7f9b08036f10 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.057+0000 7f9b2902c700 1 --2- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 0x7f9b080393d0 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c110 tx=0x7f9b20009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T10:59:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.061+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9b241227a0 con 0x7f9b241366d0 2024-09-16T10:59:58.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.061+0000 7f9b15ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T10:59:58.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.061+0000 7f9b15ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T10:59:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b2a02e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T10:59:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b2a02e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b2a02e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T10:59:58.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9b241227a0 con 0x7f9b241366d0 2024-09-16T10:59:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b15ffb700 1 -- 172.21.15.5:0/3821512999 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f9b241227a0 con 0x7f9b241366d0 2024-09-16T10:59:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b15ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T10:59:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.297+0000 7f9b15ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T10:59:58.305 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 10 librados: watch_flush enter 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 10 librados: watch_flush exit 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 20 client.74166.objecter shutdown clearing up homeless session... 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 10 client.74166.objecter successfully canceled tick 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 msgr2=0x7f9b080393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 0x7f9b080393d0 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000c110 tx=0x7f9b20009000 comp rx=0 tx=0).stop 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 10 monclient: shutdown 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 msgr2=0x7f9b241ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800b9e0 tx=0x7f9b1800bcf0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 shutdown_connections 2024-09-16T10:59:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9b241cae40 0x7f9b241cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9b08036f10 0x7f9b080393d0 unknown :-1 s=CLOSED pgs=3829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b241366d0 0x7f9b241ca900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 --2- 172.21.15.5:0/3821512999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b241c7400 0x7f9b24136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 >> 172.21.15.5:0/3821512999 conn(0x7f9b240a4d80 msgr2=0x7f9b2413a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 shutdown_connections 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 -- 172.21.15.5:0/3821512999 wait complete. 2024-09-16T10:59:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T10:59:58.301+0000 7f9b2a02e700 1 librados: shutdown 2024-09-16T11:00:01.321 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:00:01.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: get_monmap_and_config 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: build_initial_monmap 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: monmap: 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:00:01.438915+0000 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:00:01.438915+0000 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:00:01.440 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:00:01.441 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:00:01.441 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 Processor -- start 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 -- start start 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: init 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc9009ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-b con 0x7efc9009ea30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc9009cb20 addr v1:172.21.15.143:6789/0 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc9009f350 addr v1:172.21.15.5:6789/0 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9009fb90 con 0x7efc9009f350 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc9013ae20 con 0x7efc9009cb20 2024-09-16T11:00:01.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9009cf00 con 0x7efc9009ea30 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9808e700 10 monclient(hunting): authenticate will time out at 1460.015484s 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc95e2a700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 msgr2=0x7efc9009ee10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc95e2a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc9009ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9662b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7efc9009f350 0x7efc9013a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:46632/0 (socket says 172.21.15.5:46632) 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.433+0000 7efc9662b700 1 -- 172.21.15.5:0/2955227831 learned_addr learned my addr 172.21.15.5:0/2955227831 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633152091 0 0) 0x7efc9009fb90 con 0x7efc9009f350 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): _negotiate 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): my global_id is 74169 2024-09-16T11:00:01.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c0035d0 con 0x7efc9009f350 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653938012 0 0) 0x7efc7c0035d0 con 0x7efc9009f350 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 msgr2=0x7efc9009ee10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc9009ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 >> v1:172.21.15.143:6789/0 conn(0x7efc9009cb20 legacy=0x7efc9013a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _finish_hunting 0 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 monclient: found mon.noname-a-legacy 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T11:00:01.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9009fdf0 con 0x7efc9009f350 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _finish_auth 0 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _check_auth_tickets 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7efc88001a10 con 0x7efc9009f350 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: dump: 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.455 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:01.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 >> v1:172.21.15.5:6789/0 conn(0x7efc9009f350 legacy=0x7efc9013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc7c003a50 0x7efc7c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): picked mon.a con 0x7efc7c003a50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c004870 0x7efc7c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): picked mon.b con 0x7efc7c004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc7c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): picked mon.c con 0x7efc9009ea30 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c005580 con 0x7efc7c003a50 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0056f0 con 0x7efc7c004870 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c00bf70 con 0x7efc9009ea30 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9662b700 1 -- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 msgr2=0x7efc7c00ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9662b700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc7c00ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc7c003a50 0x7efc7c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): get_auth_request con 0x7efc7c003a50 auth_method 0 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:01.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c004870 0x7efc7c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 10 monclient(hunting): get_auth_request con 0x7efc7c004870 auth_method 0 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient(hunting): handle_auth_done global_id 74169 payload 293 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 -- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 msgr2=0x7efc7c00ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc9009ea30 0x7efc7c00ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 -- 172.21.15.5:0/2955227831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c004870 msgr2=0x7efc7c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c004870 0x7efc7c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient: _finish_hunting 0 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 monclient: found mon.a 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 -- 172.21.15.5:0/2955227831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0035d0 con 0x7efc7c003a50 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient: _finish_auth 0 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 10 monclient: _check_auth_tickets 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95629700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c004870 0x7efc7c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:00:01.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 5 monclient: authenticate success, global_id 74169 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc95e2a700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc7c003a50 0x7efc7c003e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc80009e30 tx=0x7efc7c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5103393a11730a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc800098d0 con 0x7efc7c003a50 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: dump: 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.461 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8000bd60 con 0x7efc7c003a50 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 10 monclient: get_monmap_and_config success 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 4 set_mon_vals no callback set 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 1 -- 172.21.15.5:0/2955227831 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc80003ca0 con 0x7efc7c003a50 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc94e28700 10 monclient: dump: 2024-09-16T11:00:01.462 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 10 monclient: shutdown 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 -- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc7c003a50 msgr2=0x7efc7c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 --2- 172.21.15.5:0/2955227831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc7c003a50 0x7efc7c003e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efc80009e30 tx=0x7efc7c0058a0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9662b700 1 -- 172.21.15.5:0/2955227831 reap_dead start 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 -- 172.21.15.5:0/2955227831 shutdown_connections 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 -- 172.21.15.5:0/2955227831 >> 172.21.15.5:0/2955227831 conn(0x7efc900a0b30 msgr2=0x7efc900a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 -- 172.21.15.5:0/2955227831 shutdown_connections 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 1 -- 172.21.15.5:0/2955227831 wait complete. 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 10 monclient: build_initial_monmap 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.437+0000 7efc9808e700 10 monclient: monmap: 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:00:01.464 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 librados: starting msgr at 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 librados: starting objecter 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 Processor -- start 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- start start 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 librados: setting wanted keys 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 librados: calling monclient init 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: init 2024-09-16T11:00:01.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 0x7efc901cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-c con 0x7efc901cf4a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc901cfdc0 0x7efc90126750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-b con 0x7efc901cfdc0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): picked mon.noname-a con 0x7efc90126c90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc90070540 con 0x7efc90126c90 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9009b1b0 con 0x7efc901cfdc0 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc90129a60 con 0x7efc901cf4a0 2024-09-16T11:00:01.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient(hunting): authenticate will time out at 1460.021538s 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95e2a700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 msgr2=0x7efc901cf880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95e2a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 0x7efc901cf880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35654/0 (socket says 172.21.15.5:35654) 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 -- 172.21.15.5:0/534344352 learned_addr learned my addr 172.21.15.5:0/534344352 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): get_auth_request con 0x7efc90126c90 auth_method 0 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc901cfdc0 0x7efc90126750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:01.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): get_auth_request con 0x7efc901cfdc0 auth_method 0 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc95629700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient(hunting): handle_auth_done global_id 74172 payload 824 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 -- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 msgr2=0x7efc901cf880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 0x7efc901cf880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 -- 172.21.15.5:0/534344352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc901cfdc0 msgr2=0x7efc90126750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc901cfdc0 0x7efc90126750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient: _finish_hunting 0 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 monclient: found mon.noname-a 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 -- 172.21.15.5:0/534344352 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc900a3430 con 0x7efc90126c90 2024-09-16T11:00:01.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9662b700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efc9009f730 tx=0x7efc8800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 client.?.objecter ms_handle_connect 0x7efc90126c90 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8800a930 con 0x7efc90126c90 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: dump: 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.470 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: _finish_auth 0 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: _check_auth_tickets 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 5 monclient: authenticate success, global_id 74172 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _renew_subs 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8800aab0 con 0x7efc90126c90 2024-09-16T11:00:01.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc90128e30 con 0x7efc90126c90 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8800ae60 con 0x7efc90126c90 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 client.74172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc87fff700 4 set_mon_vals no callback set 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc867fc700 10 monclient: dump: 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.473 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _renew_subs 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9012a630 con 0x7efc90126c90 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.441+0000 7efc9808e700 1 librados: init done 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7efc8801c3b0 con 0x7efc90126c90 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc9808e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:01.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc9808e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc9808e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 1 --2- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 0x7efc740393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9012a190 con 0x7efc90126c90 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(401..401 src has 344..401) v4 ==== 5471+0+0 (secure 0 0 0) 0x7efc88047720 con 0x7efc90126c90 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 10 client.74172.objecter ms_dispatch 0x7efc90000fd0 osd_map(401..401 src has 344..401) v4 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 3 client.74172.objecter handle_osd_map got epochs [401,401] > 0 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 3 client.74172.objecter handle_osd_map decoding full epoch 401 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 20 client.74172.objecter dump_active .. 0 homeless 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc95e2a700 1 --2- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 0x7efc740393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc95e2a700 10 monclient: get_auth_request con 0x7efc74036f10 auth_method 0 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc867fc700 10 client.74172.objecter ms_handle_connect 0x7efc74036f10 2024-09-16T11:00:01.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.445+0000 7efc95e2a700 1 --2- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 0x7efc740393d0 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7efc90071fe0 tx=0x7efc80019040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:01.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.449+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efc8801a6d0 con 0x7efc90126c90 2024-09-16T11:00:01.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.449+0000 7efc867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:01.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.449+0000 7efc867fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:00:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.685+0000 7efc9808e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:00:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.685+0000 7efc9808e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.685+0000 7efc9808e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:01.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.685+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efc900a1950 con 0x7efc90126c90 2024-09-16T11:00:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc867fc700 1 -- 172.21.15.5:0/534344352 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7efc88018070 con 0x7efc90126c90 2024-09-16T11:00:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:01.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc867fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:00:01.694 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc9808e700 10 librados: watch_flush enter 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc9808e700 10 librados: watch_flush exit 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc9808e700 20 client.74172.objecter shutdown clearing up homeless session... 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.689+0000 7efc9808e700 10 client.74172.objecter successfully canceled tick 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 msgr2=0x7efc740393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 0x7efc740393d0 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7efc90071fe0 tx=0x7efc80019040 comp rx=0 tx=0).stop 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 10 monclient: shutdown 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:01.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 msgr2=0x7efc90127110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efc9009f730 tx=0x7efc8800ce70 comp rx=0 tx=0).stop 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 shutdown_connections 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> v2:172.21.15.143:6812/1098205034 conn(0x7efc74036f10 0x7efc740393d0 unknown :-1 s=CLOSED pgs=3830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc90126c90 0x7efc90127110 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc901cfdc0 0x7efc90126750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 --2- 172.21.15.5:0/534344352 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7efc901cf4a0 0x7efc901cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 >> 172.21.15.5:0/534344352 conn(0x7efc900a0b30 msgr2=0x7efc9013a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 shutdown_connections 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 -- 172.21.15.5:0/534344352 wait complete. 2024-09-16T11:00:01.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:01.693+0000 7efc9808e700 1 librados: shutdown 2024-09-16T11:00:04.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:00:04.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 10 monclient: get_monmap_and_config 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 10 monclient: build_initial_monmap 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 10 monclient: monmap: 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:00:04.831638+0000 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:00:04.831638+0000 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:00:04.833 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:00:04.834 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:00:04.834 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:00:04.834 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 1 Processor -- start 2024-09-16T11:00:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 1 -- start start 2024-09-16T11:00:04.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.825+0000 7fd51cfe7700 10 monclient: init 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5180a2bf0 addr v1:172.21.15.5:6789/0 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd51813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-b con 0x7fd51809f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd5180a36e0 0x7fd51813d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-a con 0x7fd5180a36e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180a3c40 con 0x7fd5180a2bf0 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180a4300 con 0x7fd5180a36e0 2024-09-16T11:00:04.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51809fda0 con 0x7fd51809f780 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient(hunting): authenticate will time out at 1463.408662s 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd516ffd700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 msgr2=0x7fd51813ae00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd516ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd51813ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd5180a36e0 0x7fd51813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd5180a36e0 0x7fd51813d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60368/0 (socket says 172.21.15.5:60368) 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 -- 172.21.15.5:0/3623185523 learned_addr learned my addr 172.21.15.5:0/3623185523 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): get_auth_request con 0x7fd5180a36e0 auth_method 0 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:04.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5177fe700 1 --1- 172.21.15.5:0/3623185523 >> v1:172.21.15.5:6789/0 conn(0x7fd5180a2bf0 0x7fd5180a2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54926/0 (socket says 172.21.15.5:54926) 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient(hunting): handle_auth_done global_id 74175 payload 274 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 -- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 msgr2=0x7fd51813ae00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 --2- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd51813ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 -- 172.21.15.5:0/3623185523 >> v1:172.21.15.5:6789/0 conn(0x7fd5180a2bf0 legacy=0x7fd5180a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient: _finish_hunting 0 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 monclient: found mon.noname-a 2024-09-16T11:00:04.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 -- 172.21.15.5:0/3623185523 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180a0080 con 0x7fd5180a36e0 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd517fff700 1 --2- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd510009e50 tx=0x7fd51000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca5211327aaa44d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 1 -- 172.21.15.5:0/3623185523 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000e070 con 0x7fd5180a36e0 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: dump: 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:04.841 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: _finish_auth 0 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: _check_auth_tickets 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 5 monclient: authenticate success, global_id 74175 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 1 -- 172.21.15.5:0/3623185523 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51000b9c0 con 0x7fd5180a36e0 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient: get_monmap_and_config success 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 4 set_mon_vals no callback set 2024-09-16T11:00:04.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 1 -- 172.21.15.5:0/3623185523 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000bd70 con 0x7fd5180a36e0 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd5167fc700 10 monclient: dump: 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:04.843 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 10 monclient: shutdown 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 msgr2=0x7fd51813d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813d350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd510009e50 tx=0x7fd51000b330 comp rx=0 tx=0).stop 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- 172.21.15.5:0/3623185523 shutdown_connections 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813d350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/3623185523 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd51813ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- 172.21.15.5:0/3623185523 >> 172.21.15.5:0/3623185523 conn(0x7fd5180a53e0 msgr2=0x7fd518073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- 172.21.15.5:0/3623185523 shutdown_connections 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.829+0000 7fd51cfe7700 1 -- 172.21.15.5:0/3623185523 wait complete. 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient: build_initial_monmap 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient: monmap: 2024-09-16T11:00:04.844 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:00:04.845 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 librados: starting msgr at 2024-09-16T11:00:04.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 librados: starting objecter 2024-09-16T11:00:04.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 Processor -- start 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 -- start start 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 librados: setting wanted keys 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 librados: calling monclient init 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient: init 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:04.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd518135410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-c con 0x7fd51809f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5180a2bf0 0x7fd518135950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-b con 0x7fd5180a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): picked mon.noname-a con 0x7fd5180a36e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd5177fe700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 msgr2=0x7fd518135410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51811e9f0 con 0x7fd5180a36e0 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51811ef70 con 0x7fd5180a2bf0 2024-09-16T11:00:04.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180a7340 con 0x7fd51809f780 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd5177fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd518135410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd51cfe7700 10 monclient(hunting): authenticate will time out at 1463.414545s 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60380/0 (socket says 172.21.15.5:60380) 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 -- 172.21.15.5:0/1125006738 learned_addr learned my addr 172.21.15.5:0/1125006738 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): get_auth_request con 0x7fd5180a36e0 auth_method 0 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5180a2bf0 0x7fd518135950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:04.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): get_auth_request con 0x7fd5180a2bf0 auth_method 0 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:04.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd516ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient(hunting): handle_auth_done global_id 74178 payload 824 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 -- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 msgr2=0x7fd518135410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd518135410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 -- 172.21.15.5:0/1125006738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5180a2bf0 msgr2=0x7fd518135950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5180a2bf0 0x7fd518135950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient: _finish_hunting 0 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 monclient: found mon.noname-a 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 -- 172.21.15.5:0/1125006738 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510009b30 con 0x7fd5180a36e0 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd4fffff700 10 client.?.objecter ms_handle_connect 0x7fd5180a36e0 2024-09-16T11:00:04.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd517fff700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd51000b730 tx=0x7fd510004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.833+0000 7fd4fffff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510004920 con 0x7fd5180a36e0 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: dump: 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:04.857 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: _finish_auth 0 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: _check_auth_tickets 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 5 monclient: authenticate success, global_id 74178 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _renew_subs 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5100092e0 con 0x7fd5180a36e0 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51809f4e0 con 0x7fd5180a36e0 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:04.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 client.74178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _renew_subs 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd515ffb700 4 set_mon_vals no callback set 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd518120330 con 0x7fd5180a36e0 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51001c660 con 0x7fd5180a36e0 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 1 librados: init done 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:04.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 monclient: dump: 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5180a2fd0 con 0x7fd5180a36e0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd51001c7c0 con 0x7fd5180a36e0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 --2- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 0x7fd504039400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(401..401 src has 344..401) v4 ==== 5471+0+0 (secure 0 0 0) 0x7fd510049c00 con 0x7fd5180a36e0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 10 client.74178.objecter ms_dispatch 0x7fd518000fd0 osd_map(401..401 src has 344..401) v4 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 3 client.74178.objecter handle_osd_map got epochs [401,401] > 0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 3 client.74178.objecter handle_osd_map decoding full epoch 401 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd4fffff700 20 client.74178.objecter dump_active .. 0 homeless 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd5177fe700 1 --2- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 0x7fd504039400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.837+0000 7fd5177fe700 10 monclient: get_auth_request con 0x7fd504036f40 auth_method 0 2024-09-16T11:00:04.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.841+0000 7fd4fffff700 10 client.74178.objecter ms_handle_connect 0x7fd504036f40 2024-09-16T11:00:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.841+0000 7fd5177fe700 1 --2- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 0x7fd504039400 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fd50c001120 tx=0x7fd50c007ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.845+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd51001ca70 con 0x7fd5180a36e0 2024-09-16T11:00:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.845+0000 7fd4fffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:04.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:04.845+0000 7fd4fffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:00:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd51cfe7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:00:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd51cfe7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd51cfe7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:05.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd518120080 con 0x7fd5180a36e0 2024-09-16T11:00:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd4fffff700 1 -- 172.21.15.5:0/1125006738 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fd510023360 con 0x7fd5180a36e0 2024-09-16T11:00:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd4fffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:05.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.077+0000 7fd4fffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:00:05.084 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:00:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 10 librados: watch_flush enter 2024-09-16T11:00:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 10 librados: watch_flush exit 2024-09-16T11:00:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 20 client.74178.objecter shutdown clearing up homeless session... 2024-09-16T11:00:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 10 client.74178.objecter successfully canceled tick 2024-09-16T11:00:05.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 msgr2=0x7fd504039400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 0x7fd504039400 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fd50c001120 tx=0x7fd50c007ad0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 10 monclient: shutdown 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 msgr2=0x7fd51813aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd51000b730 tx=0x7fd510004af0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 shutdown_connections 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd504036f40 0x7fd504039400 unknown :-1 s=CLOSED pgs=3831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd5180a36e0 0x7fd51813aa70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5180a2bf0 0x7fd518135950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 --2- 172.21.15.5:0/1125006738 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd51809f780 0x7fd518135410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 >> 172.21.15.5:0/1125006738 conn(0x7fd5180a53e0 msgr2=0x7fd51806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:05.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 shutdown_connections 2024-09-16T11:00:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 -- 172.21.15.5:0/1125006738 wait complete. 2024-09-16T11:00:05.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:05.081+0000 7fd51cfe7700 1 librados: shutdown 2024-09-16T11:00:08.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:00:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.213+0000 7f721ffff700 10 monclient: get_monmap_and_config 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.213+0000 7f721ffff700 10 monclient: build_initial_monmap 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient: monmap: 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:00:08.220848+0000 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:00:08.220848+0000 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:00:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:00:08.223 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:00:08.223 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:00:08.223 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:08.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 Processor -- start 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 -- start start 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient: init 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f722009b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f722009bd00 addr v1:172.21.15.5:6789/0 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f722009c110 addr v1:172.21.15.143:6789/0 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72201334d0 con 0x7f722009bd00 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722009c560 con 0x7f722009c110 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722009e350 con 0x7f722009b3e0 2024-09-16T11:00:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721ffff700 10 monclient(hunting): authenticate will time out at 1466.797201s 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721e7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f722009bd00 0x7f72201326b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54930/0 (socket says 172.21.15.5:54930) 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 learned_addr learned my addr 172.21.15.5:0/3639388121 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:08.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.217+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:08.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:08.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.421+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:08.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:08.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:08.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:08.825+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:00:09.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:09.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:09.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:09.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:09.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:09.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:09.628+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): tick 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 monclient(hunting): continuing hunt 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 msgr2=0x7f722009b7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f722009b7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 >> v1:172.21.15.143:6789/0 conn(0x7f722009c110 legacy=0x7f7220132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 >> v1:172.21.15.5:6789/0 conn(0x7f722009bd00 legacy=0x7f72201326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient: _add_conns ranks=[2,4] 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): picked mon.noname-b con 0x7f720c0059f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): picked mon.noname-c con 0x7f720c008550 addr v2:172.21.15.143:3300/0 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722009e350 con 0x7f720c008550 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c00b090 con 0x7f720c0059f0 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721cff9700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 msgr2=0x7f720c007e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:11.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 10 monclient(hunting): get_auth_request con 0x7f720c008550 auth_method 0 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:11.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.216+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:11.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 msgr2=0x7f720c007e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:11.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 10 monclient(hunting): get_auth_request con 0x7f720c008550 auth_method 0 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:11.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.420+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:00:11.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 msgr2=0x7f720c007e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:11.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:00:11.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 10 monclient(hunting): get_auth_request con 0x7f720c008550 auth_method 0 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:11.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:11.824+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:00:12.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 msgr2=0x7f720c007e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:12.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 10 monclient(hunting): get_auth_request con 0x7f720c008550 auth_method 0 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:00:12.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:00:12.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:12.628+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:00:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): tick 2024-09-16T11:00:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): _check_auth_tickets 2024-09-16T11:00:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 monclient(hunting): continuing hunt 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 msgr2=0x7f720c007e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c0059f0 0x7f720c007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 msgr2=0x7f720c00a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c008550 0x7f720c00a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721f7fe700 1 -- 172.21.15.5:0/3639388121 reap_dead start 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f720c00d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): picked mon.noname-a con 0x7f722009b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 0x7f720c0089c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): picked mon.noname-c con 0x7f720c005e10 addr v2:172.21.15.143:3300/0 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f720c009370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): picked mon.noname-b con 0x7f720c008f00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722009e350 con 0x7f722009b3e0 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720c00b090 con 0x7f720c005e10 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 1 -- 172.21.15.5:0/3639388121 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c009930 con 0x7f720c008f00 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721cff9700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721f7fe700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 msgr2=0x7f720c009370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721f7fe700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f720c009370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.5:3300/0 conn(0x7f722009b3e0 0x7f720c00d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 0x7f720c0089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): get_auth_request con 0x7f720c005e10 auth_method 0 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient(hunting): handle_auth_done global_id 74181 payload 274 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 msgr2=0x7f720c009370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f720c009370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 msgr2=0x7f720c0089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 0x7f720c0089c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 monclient: found mon.noname-a 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 -- 172.21.15.5:0/3639388121 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722009ca20 con 0x7f722009b3e0 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721e7fc700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 0x7f720c0089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:00:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.216+0000 7f721effd700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f720c00d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72200685d0 tx=0x7f7210009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2108db9b1524c15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 1 -- 172.21.15.5:0/3639388121 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210004850 con 0x7f722009b3e0 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: dump: 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 1 -- 172.21.15.5:0/3639388121 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7210017470 con 0x7f722009b3e0 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 5 monclient: authenticate success, global_id 74181 2024-09-16T11:00:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 10 monclient: get_monmap_and_config success 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 1 -- 172.21.15.5:0/3639388121 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210017820 con 0x7f722009b3e0 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 4 set_mon_vals no callback set 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721dffb700 10 monclient: dump: 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 10 monclient: shutdown 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 -- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 msgr2=0x7f720c00d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f720c00d4a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72200685d0 tx=0x7f7210009f80 comp rx=0 tx=0).stop 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 -- 172.21.15.5:0/3639388121 shutdown_connections 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f720c009370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 --2- 172.21.15.5:0/3639388121 >> v2:172.21.15.143:3300/0 conn(0x7f720c005e10 0x7f720c0089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 --2- 172.21.15.5:0/3639388121 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f720c00d4a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 -- 172.21.15.5:0/3639388121 >> 172.21.15.5:0/3639388121 conn(0x7f72200a45d0 msgr2=0x7f72200a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 -- 172.21.15.5:0/3639388121 shutdown_connections 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 1 -- 172.21.15.5:0/3639388121 wait complete. 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 10 monclient: build_initial_monmap 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.220+0000 7f721ffff700 10 monclient: monmap: 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:00:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 librados: starting msgr at 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 librados: starting objecter 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 Processor -- start 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 -- start start 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 librados: setting wanted keys 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 librados: calling monclient init 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient: init 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 0x7f72201300a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f720c005e10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f7220120810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f720c008f00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f722009b3e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72201230f0 con 0x7f722009b3e0 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7220132f80 con 0x7f720c005e10 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7220124810 con 0x7f720c008f00 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721ffff700 10 monclient(hunting): authenticate will time out at 1472.806161s 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721e7fc700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 msgr2=0x7f7220120810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721e7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f7220120810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 0x7f72201300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37612/0 (socket says 172.21.15.5:37612) 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 1 -- 172.21.15.5:0/964913861 learned_addr learned my addr 172.21.15.5:0/964913861 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): get_auth_request con 0x7f722009b3e0 auth_method 0 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): get_auth_request con 0x7f720c005e10 auth_method 0 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient(hunting): handle_auth_done global_id 64288 payload 824 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 -- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 msgr2=0x7f7220120810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f7220120810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 -- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 msgr2=0x7f7220120d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 monclient: found mon.noname-b 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721effd700 1 -- 172.21.15.5:0/964913861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721400f040 con 0x7f720c005e10 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.224+0000 7f721f7fe700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721effd700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 0x7f72201300a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f722009d770 tx=0x7f7210004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 client.?.objecter ms_handle_connect 0x7f720c005e10 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210009410 con 0x7f720c005e10 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: dump: 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 5 monclient: authenticate success, global_id 64288 2024-09-16T11:00:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7210020620 con 0x7f720c005e10 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _renew_subs 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7210009900 con 0x7f720c005e10 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72100209b0 con 0x7f720c005e10 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 client.64288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 monclient: dump: 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721d7fa700 4 set_mon_vals no callback set 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _renew_subs 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722012f490 con 0x7f720c005e10 2024-09-16T11:00:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 1 librados: init done 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7210020b10 con 0x7f720c005e10 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722011cd00 con 0x7f720c005e10 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 --2- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 0x7f720c03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(403..403 src has 344..403) v4 ==== 4932+0+0 (secure 0 0 0) 0x7f721004a0a0 con 0x7f720c005e10 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 10 client.64288.objecter ms_dispatch 0x7f7220000fd0 osd_map(403..403 src has 344..403) v4 2024-09-16T11:00:14.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 3 client.64288.objecter handle_osd_map got epochs [403,403] > 0 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 3 client.64288.objecter handle_osd_map decoding full epoch 403 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f7207fff700 20 client.64288.objecter dump_active .. 0 homeless 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721e7fc700 1 --2- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 0x7f720c03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.228+0000 7f721e7fc700 10 monclient: get_auth_request con 0x7f720c0388b0 auth_method 0 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.232+0000 7f7207fff700 10 client.64288.objecter ms_handle_connect 0x7f720c0388b0 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.232+0000 7f721e7fc700 1 --2- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 0x7f720c03ad70 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7f7214000c00 tx=0x7f7214009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.240+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7210020dc0 con 0x7f720c005e10 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.240+0000 7f7207fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.240+0000 7f7207fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f721ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f721ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f721ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7220124100 con 0x7f720c005e10 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f7207fff700 1 -- 172.21.15.5:0/964913861 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f721001b020 con 0x7f720c005e10 2024-09-16T11:00:14.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f7207fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:00:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.476+0000 7f7207fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:00:14.485 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 10 librados: watch_flush enter 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 10 librados: watch_flush exit 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 20 client.64288.objecter shutdown clearing up homeless session... 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 10 client.64288.objecter successfully canceled tick 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 msgr2=0x7f720c03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 0x7f720c03ad70 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7f7214000c00 tx=0x7f7214009040 comp rx=0 tx=0).stop 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 10 monclient: shutdown 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 msgr2=0x7f72201300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 0x7f72201300a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f722009d770 tx=0x7f7210004af0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 shutdown_connections 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> v2:172.21.15.143:6812/1098205034 conn(0x7f720c0388b0 0x7f720c03ad70 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f722009b3e0 0x7f7220120d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f720c008f00 0x7f7220120810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 --2- 172.21.15.5:0/964913861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f720c005e10 0x7f72201300a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 >> 172.21.15.5:0/964913861 conn(0x7f72200a45d0 msgr2=0x7f72200a2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 shutdown_connections 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 -- 172.21.15.5:0/964913861 wait complete. 2024-09-16T11:00:14.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.480+0000 7f721ffff700 1 librados: shutdown 2024-09-16T11:00:14.500 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T11:00:14.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T11:00:14.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:00:14.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: get_monmap_and_config 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: build_initial_monmap 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: monmap: 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:00:14.616923+0000 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:00:14.616923+0000 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:00:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:00:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:00:14.619 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 Processor -- start 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- start start 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: init 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a80a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd3a80a2cf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3a80a3e90 addr v1:172.21.15.143:6789/0 2024-09-16T11:00:14.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3a80a3610 0x7fd3a81361c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd3a80a3610 addr v2:172.21.15.5:3300/0 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a80a4a80 con 0x7fd3a80a3e90 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80a4310 con 0x7fd3a80a3610 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a06c0 con 0x7fd3a80a2cf0 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient(hunting): authenticate will time out at 1473.193260s 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a659c700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 msgr2=0x7fd3a80a30d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a80a30d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3a80a3610 0x7fd3a81361c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3a80a3610 0x7fd3a81361c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37626/0 (socket says 172.21.15.5:37626) 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/3598791840 learned_addr learned my addr 172.21.15.5:0/3598791840 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): get_auth_request con 0x7fd3a80a3610 auth_method 0 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_done global_id 74187 payload 274 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 msgr2=0x7fd3a80a30d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 --2- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a80a30d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/3598791840 >> v1:172.21.15.143:6789/0 conn(0x7fd3a80a3e90 legacy=0x7fd3a8135ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 monclient: found mon.noname-a 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/3598791840 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80a03e0 con 0x7fd3a80a3610 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a6d9d700 1 --2- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3610 0x7fd3a81361c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009910 tx=0x7fd3a0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ff3813c97c424d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 1 -- 172.21.15.5:0/3598791840 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0004230 con 0x7fd3a80a3610 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: dump: 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.625 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 1 -- 172.21.15.5:0/3598791840 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a0004390 con 0x7fd3a80a3610 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 5 monclient: authenticate success, global_id 74187 2024-09-16T11:00:14.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: get_monmap_and_config success 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 4 set_mon_vals no callback set 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 1 -- 172.21.15.5:0/3598791840 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0009e60 con 0x7fd3a80a3610 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a559a700 10 monclient: dump: 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.627 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 10 monclient: shutdown 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3610 msgr2=0x7fd3a81361c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 --2- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3610 0x7fd3a81361c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009910 tx=0x7fd3a0009c20 comp rx=0 tx=0).stop 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- 172.21.15.5:0/3598791840 shutdown_connections 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 --2- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3610 0x7fd3a81361c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 --2- 172.21.15.5:0/3598791840 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a80a30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- 172.21.15.5:0/3598791840 >> 172.21.15.5:0/3598791840 conn(0x7fd3a80a89e0 msgr2=0x7fd3a80a4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- 172.21.15.5:0/3598791840 shutdown_connections 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.612+0000 7fd3a759e700 1 -- 172.21.15.5:0/3598791840 wait complete. 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: build_initial_monmap 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: monmap: 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:00:14.629 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 librados: starting msgr at 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 librados: starting objecter 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 Processor -- start 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- start start 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 librados: setting wanted keys 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 librados: calling monclient init 2024-09-16T11:00:14.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: init 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a81ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd3a80a2cf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a80a3610 0x7fd3a8135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd3a80a3610 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 0x7fd3a81360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a659c700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 msgr2=0x7fd3a81ca3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:14.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd3a80a3e90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81ca8f0 con 0x7fd3a80a3e90 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a81ca3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8122cd0 con 0x7fd3a80a3610 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a78d0 con 0x7fd3a80a2cf0 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient(hunting): authenticate will time out at 1473.197509s 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 0x7fd3a81360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 0x7fd3a81360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37638/0 (socket says 172.21.15.5:37638) 2024-09-16T11:00:14.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 learned_addr learned my addr 172.21.15.5:0/2668086370 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): get_auth_request con 0x7fd3a80a3e90 auth_method 0 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a80a3610 0x7fd3a8135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): get_auth_request con 0x7fd3a80a3610 auth_method 0 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient(hunting): handle_auth_done global_id 74190 payload 824 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 msgr2=0x7fd3a81ca3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd3a80a2cf0 0x7fd3a81ca3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a80a3610 msgr2=0x7fd3a8135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a80a3610 0x7fd3a8135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 monclient: found mon.noname-a 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0009570 con 0x7fd3a80a3e90 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 client.?.objecter ms_handle_connect 0x7fd3a80a3e90 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a6d9d700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 0x7fd3a81360f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0004000 tx=0x7fd3a0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00038c0 con 0x7fd3a80a3e90 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: dump: 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.641 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a0003a20 con 0x7fd3a80a3e90 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 5 monclient: authenticate success, global_id 74190 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _renew_subs 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a809f4e0 con 0x7fd3a80a3e90 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0003dd0 con 0x7fd3a80a3e90 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd396ffd700 10 monclient: dump: 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.643 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a4d99700 4 set_mon_vals no callback set 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 client.74190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _renew_subs 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8121270 con 0x7fd3a80a3e90 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 1 librados: init done 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.616+0000 7fd3a759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a759e700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 0x7fd3a811f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8067380 con 0x7fd3a811f610 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 0x7fd3a811f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient: get_auth_request con 0x7fd3a811f610 auth_method 0 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd3a0021070 con 0x7fd3a80a3e90 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 --2- 172.21.15.5:0/2668086370 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd388036f10 0x7fd3880393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 10 monclient(hunting): handle_auth_done global_id 74193 payload 824 2024-09-16T11:00:14.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(403..403 src has 344..403) v4 ==== 4932+0+0 (secure 0 0 0) 0x7fd3a00493c0 con 0x7fd3a80a3e90 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 client.74190.objecter ms_dispatch 0x7fd3a8000fd0 osd_map(403..403 src has 344..403) v4 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 3 client.74190.objecter handle_osd_map got epochs [403,403] > 0 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 3 client.74190.objecter handle_osd_map decoding full epoch 403 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3a8121c20 con 0x7fd3a811f610 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 20 client.74190.objecter dump_active .. 0 homeless 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a5d9b700 1 --2- 172.21.15.5:0/2668086370 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd388036f10 0x7fd3880393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 client.74190.objecter ms_handle_connect 0x7fd3a811f610 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 client.74190.objecter resend_mon_ops 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a5d9b700 10 monclient: get_auth_request con 0x7fd388036f10 auth_method 0 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a659c700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 0x7fd3a811f9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39800ce50 con 0x7fd3a811f610 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 monclient: dump: 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 client.74190.objecter ms_handle_connect 0x7fd388036f10 2024-09-16T11:00:14.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd3a5d9b700 1 --2- 172.21.15.5:0/2668086370 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd388036f10 0x7fd3880393d0 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8122bc0 tx=0x7fd39c007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd398015470 con 0x7fd3a811f610 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 msgr2=0x7fd3a811f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 0x7fd3a811f9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398009ed0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 msgr2=0x7fd3a811f9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.620+0000 7fd396ffd700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811f610 0x7fd3a811f9f0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398009ed0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 0x7fd3a806cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a2660 con 0x7fd3a811ff30 2024-09-16T11:00:14.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 0x7fd3a806cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient: get_auth_request con 0x7fd3a811ff30 auth_method 0 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 10 monclient(hunting): handle_auth_done global_id 74196 payload 824 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 1 -- 172.21.15.5:0/2668086370 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd3a80a30d0 con 0x7fd3a811ff30 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 client.74190.objecter ms_handle_connect 0x7fd3a811ff30 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 client.74190.objecter resend_mon_ops 2024-09-16T11:00:14.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a659c700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 0x7fd3a806cae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39801b9b0 con 0x7fd3a811ff30 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 monclient: dump: 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.650 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7fd39801bb10 con 0x7fd3a811ff30 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 msgr2=0x7fd3a806cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 0x7fd3a806cae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398005e50 comp rx=0 tx=0).stop 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 msgr2=0x7fd3a806cae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd396ffd700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a811ff30 0x7fd3a806cae0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd39800bc60 tx=0x7fd398005e50 comp rx=0 tx=0).stop 2024-09-16T11:00:14.651 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 46, 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 1, 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:00:14.652 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:00:14.653 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:00:14.654 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:00:14.655 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.659 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.660 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.661 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.662 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:00:14.663 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.664 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:00:14.665 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.665 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.665 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.665 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:00:14.665 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 librados: watch_flush enter 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 librados: watch_flush exit 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 20 client.74190.objecter shutdown clearing up homeless session... 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 client.74190.objecter successfully canceled tick 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd388036f10 msgr2=0x7fd3880393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 --2- 172.21.15.5:0/2668086370 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd388036f10 0x7fd3880393d0 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8122bc0 tx=0x7fd39c007ee0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 10 monclient: shutdown 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 msgr2=0x7fd3a81360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 --2- 172.21.15.5:0/2668086370 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3a80a3e90 0x7fd3a81360f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0004000 tx=0x7fd3a0004770 comp rx=0 tx=0).stop 2024-09-16T11:00:14.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 reap_dead start 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a6d9d700 1 -- 172.21.15.5:0/2668086370 reap_dead start 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 shutdown_connections 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 >> 172.21.15.5:0/2668086370 conn(0x7fd3a80a89e0 msgr2=0x7fd3a8070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 shutdown_connections 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 -- 172.21.15.5:0/2668086370 wait complete. 2024-09-16T11:00:14.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.628+0000 7fd3a759e700 1 librados: shutdown 2024-09-16T11:00:14.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:00:14.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T11:00:14.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:00:14.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 10 monclient: get_monmap_and_config 2024-09-16T11:00:14.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 10 monclient: build_initial_monmap 2024-09-16T11:00:14.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 10 monclient: monmap: 2024-09-16T11:00:14.787 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:00:14.786749+0000 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:00:14.786749+0000 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:00:14.788 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 1 Processor -- start 2024-09-16T11:00:14.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 1 -- start start 2024-09-16T11:00:14.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.780+0000 7f0aa359e700 10 monclient: init 2024-09-16T11:00:14.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0aa40a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0aa40a3770 0x7f0aa413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-c con 0x7f0aa40a3770 addr v2:172.21.15.143:3300/0 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0aa413b1c0 addr v1:172.21.15.5:6789/0 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40a3db0 con 0x7f0aa413b1c0 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40a4b70 con 0x7f0aa40a2c80 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa40a0850 con 0x7f0aa40a3770 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient(hunting): authenticate will time out at 1473.364356s 2024-09-16T11:00:14.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa2d9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0aa413b1c0 0x7f0aa413d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33510/0 (socket says 172.21.15.5:33510) 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa2d9d700 1 -- 172.21.15.5:0/1472795488 learned_addr learned my addr 172.21.15.5:0/1472795488 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 --2- 172.21.15.5:0/1472795488 >> v2:172.21.15.143:3300/0 conn(0x7f0aa40a3770 0x7f0aa413ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): get_auth_request con 0x7f0aa40a3770 auth_method 0 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 1 -- 172.21.15.5:0/1472795488 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968108460 0 0) 0x7f0aa40a3db0 con 0x7f0aa413b1c0 2024-09-16T11:00:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient(hunting): _negotiate 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient(hunting): my global_id is 74199 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 1 -- 172.21.15.5:0/1472795488 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a900035d0 con 0x7f0aa413b1c0 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_done global_id 64294 payload 274 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 -- 172.21.15.5:0/1472795488 >> v1:172.21.15.143:6789/0 conn(0x7f0aa40a2c80 legacy=0x7f0aa40a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 -- 172.21.15.5:0/1472795488 >> v1:172.21.15.5:6789/0 conn(0x7f0aa413b1c0 legacy=0x7f0aa413d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 monclient: found mon.noname-c 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 -- 172.21.15.5:0/1472795488 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa40a4180 con 0x7f0aa40a3770 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0a99d9b700 1 --2- 172.21.15.5:0/1472795488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa40a3770 0x7f0aa413ac80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009850 tx=0x7f0a8c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c424e7e99a37f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 1 -- 172.21.15.5:0/1472795488 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c011830 con 0x7f0aa40a3770 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: dump: 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.796 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 5 monclient: authenticate success, global_id 64294 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 1 -- 172.21.15.5:0/1472795488 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c011990 con 0x7f0aa40a3770 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: get_monmap_and_config success 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 4 set_mon_vals no callback set 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 1 -- 172.21.15.5:0/1472795488 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c011d00 con 0x7f0aa40a3770 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:00:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa1d9b700 10 monclient: dump: 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: shutdown 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- 172.21.15.5:0/1472795488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa40a3770 msgr2=0x7f0aa413ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 --2- 172.21.15.5:0/1472795488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa40a3770 0x7f0aa413ac80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009850 tx=0x7f0a8c009b60 comp rx=0 tx=0).stop 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- 172.21.15.5:0/1472795488 shutdown_connections 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 --2- 172.21.15.5:0/1472795488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa40a3770 0x7f0aa413ac80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- 172.21.15.5:0/1472795488 >> 172.21.15.5:0/1472795488 conn(0x7f0aa40a4d80 msgr2=0x7f0aa40a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- 172.21.15.5:0/1472795488 shutdown_connections 2024-09-16T11:00:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 1 -- 172.21.15.5:0/1472795488 wait complete. 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: build_initial_monmap 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.784+0000 7f0aa359e700 10 monclient: monmap: 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:00:14.806 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 librados: starting msgr at 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 librados: starting objecter 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 Processor -- start 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- start start 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 librados: setting wanted keys 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 librados: calling monclient init 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: init 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aa40a2c80 0x7f0aa4134e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-c con 0x7f0aa40a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 0x7f0aa4135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-a con 0x7f0aa40a3770 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:00:14.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa413b1c0 0x7f0aa411e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): picked mon.noname-b con 0x7f0aa413b1c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41cb5d0 con 0x7f0aa40a3770 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa411fdf0 con 0x7f0aa413b1c0 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa409f4e0 con 0x7f0aa40a2c80 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): authenticate will time out at 1473.369456s 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa259c700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aa40a2c80 msgr2=0x7f0aa4134e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aa40a2c80 0x7f0aa4134e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:00:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 0x7f0aa4135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 0x7f0aa4135350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37654/0 (socket says 172.21.15.5:37654) 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 -- 172.21.15.5:0/3563011846 learned_addr learned my addr 172.21.15.5:0/3563011846 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): get_auth_request con 0x7f0aa40a3770 auth_method 0 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa413b1c0 0x7f0aa411e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 10 monclient(hunting): get_auth_request con 0x7f0aa413b1c0 auth_method 0 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient(hunting): handle_auth_done global_id 74202 payload 824 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aa40a2c80 msgr2=0x7f0aa4134e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0aa40a2c80 0x7f0aa4134e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa413b1c0 msgr2=0x7f0aa411e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa413b1c0 0x7f0aa411e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient: _finish_hunting 0 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 monclient: found mon.noname-a 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c0094b0 con 0x7f0aa40a3770 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa2d9d700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa413b1c0 0x7f0aa411e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:00:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a99d9b700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 0x7f0aa4135350 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0090b0 tx=0x7f0a8c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f0aa40a3770 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c004470 con 0x7f0aa40a3770 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: dump: 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.811 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8c01f070 con 0x7f0aa40a3770 2024-09-16T11:00:14.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 5 monclient: authenticate success, global_id 74202 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _renew_subs 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c003900 con 0x7f0aa40a3770 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa40a7370 con 0x7f0aa40a3770 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0a9b7fe700 10 monclient: dump: 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa159a700 4 set_mon_vals no callback set 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 client.74202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _renew_subs 2024-09-16T11:00:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa4120f20 con 0x7f0aa40a3770 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 librados: init done 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 0x7f0aa411f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.788+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4067380 con 0x7f0aa411ee10 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0a8c021030 con 0x7f0aa40a3770 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 0x7f0aa411f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient: get_auth_request con 0x7f0aa411ee10 auth_method 0 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 --2- 172.21.15.5:0/3563011846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a84036f10 0x7f0a840393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(403..403 src has 344..403) v4 ==== 4932+0+0 (secure 0 0 0) 0x7f0a8c048d60 con 0x7f0aa40a3770 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 client.74202.objecter ms_dispatch 0x7f0aa4000fd0 osd_map(403..403 src has 344..403) v4 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 3 client.74202.objecter handle_osd_map got epochs [403,403] > 0 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 3 client.74202.objecter handle_osd_map decoding full epoch 403 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 10 monclient(hunting): handle_auth_done global_id 64303 payload 824 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0aa409fa10 con 0x7f0aa411ee10 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 20 client.74202.objecter dump_active .. 0 homeless 2024-09-16T11:00:14.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa2d9d700 1 --2- 172.21.15.5:0/3563011846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a84036f10 0x7f0a840393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa2d9d700 10 monclient: get_auth_request con 0x7f0a84036f10 auth_method 0 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa259c700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 0x7f0aa411f1f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000c00 tx=0x7f0a9400c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 client.74202.objecter ms_handle_connect 0x7f0aa411ee10 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 client.74202.objecter resend_mon_ops 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94005c80 con 0x7f0aa411ee10 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 monclient: dump: 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.817 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 client.74202.objecter ms_handle_connect 0x7f0a84036f10 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0aa2d9d700 1 --2- 172.21.15.5:0/3563011846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a84036f10 0x7f0a840393d0 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c001560 tx=0x7f0a9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f0a94005de0 con 0x7f0aa411ee10 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 msgr2=0x7f0aa411f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 0x7f0aa411f1f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0a94000c00 tx=0x7f0a9400c7c0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 msgr2=0x7f0aa411f1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:00:14.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.792+0000 7f0a9b7fe700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa411ee10 0x7f0aa411f1f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 0x7f0aa406cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa40a4400 con 0x7f0aa406c970 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 0x7f0aa406cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:00:14.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient: get_auth_request con 0x7f0aa406c970 auth_method 0 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 10 monclient(hunting): handle_auth_done global_id 64306 payload 824 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 1 -- 172.21.15.5:0/3563011846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0aa409f0d0 con 0x7f0aa406c970 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa259c700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 0x7f0aa406cd80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a94007fd0 tx=0x7f0a94000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 client.74202.objecter ms_handle_connect 0x7f0aa406c970 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 client.74202.objecter resend_mon_ops 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9401f070 con 0x7f0aa406c970 2024-09-16T11:00:14.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 monclient: dump: 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:00:14.822 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7f0a9401ae90 con 0x7f0aa406c970 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 msgr2=0x7f0aa406cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 0x7f0aa406cd80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a94007fd0 tx=0x7f0a94000ed0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 msgr2=0x7f0aa406cd80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0a9b7fe700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0aa406c970 0x7f0aa406cd80 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a94007fd0 tx=0x7f0a94000ed0 comp rx=0 tx=0).stop 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:00:14.823 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 46, 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.824 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.825 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:00:14.826 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:00:14.827 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.828 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.829 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:00:14.830 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.831 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.832 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:00:14.833 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:00:14.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:00:14.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:00:14.836 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:00:14.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 librados: watch_flush enter 2024-09-16T11:00:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 librados: watch_flush exit 2024-09-16T11:00:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 20 client.74202.objecter shutdown clearing up homeless session... 2024-09-16T11:00:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 client.74202.objecter successfully canceled tick 2024-09-16T11:00:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a84036f10 msgr2=0x7f0a840393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 --2- 172.21.15.5:0/3563011846 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0a84036f10 0x7f0a840393d0 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c001560 tx=0x7f0a9c009040 comp rx=0 tx=0).stop 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 10 monclient: shutdown 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 msgr2=0x7f0aa4135350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 --2- 172.21.15.5:0/3563011846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0aa40a3770 0x7f0aa4135350 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0090b0 tx=0x7f0a8c004770 comp rx=0 tx=0).stop 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa2d9d700 1 -- 172.21.15.5:0/3563011846 reap_dead start 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa2d9d700 1 -- 172.21.15.5:0/3563011846 reap_dead start 2024-09-16T11:00:14.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 shutdown_connections 2024-09-16T11:00:14.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 >> 172.21.15.5:0/3563011846 conn(0x7f0aa40a4d80 msgr2=0x7f0aa4137c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:00:14.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 shutdown_connections 2024-09-16T11:00:14.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 -- 172.21.15.5:0/3563011846 wait complete. 2024-09-16T11:00:14.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:00:14.804+0000 7f0aa359e700 1 librados: shutdown 2024-09-16T11:00:14.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:00:14.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (27085 ms) 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22983 ms) 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2909 ms) 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3021 ms) 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3009 ms) 2024-09-16T11:00:15.887 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3010 ms) 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3984 ms) 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39035 ms) 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16974 ms) 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.888 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4039 ms) 2024-09-16T11:00:15.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T11:00:15.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.889 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (25450 ms) 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24302 ms) 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15080 ms) 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4907 ms) 2024-09-16T11:00:15.892 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4024 ms) 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25098 ms) 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10100 ms) 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.893 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8853 ms) 2024-09-16T11:00:15.894 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T11:00:15.894 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:00:15.894 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (24200 ms) 2024-09-16T11:00:15.894 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T11:01:11.089 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T11:01:11.084+0000 7f35b1aee700 -1 snap_mapper.add_oid found existing snaps mapped on 60:e5823db5:test-rados-api-smithi143-24525-97::foo:31, removing 2024-09-16T11:01:11.090 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T11:01:11.088+0000 7f482b588700 -1 snap_mapper.add_oid found existing snaps mapped on 60:e5823db5:test-rados-api-smithi143-24525-97::foo:31, removing 2024-09-16T11:01:44.841 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T11:01:44.841 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T11:01:44.841 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T11:01:44.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T11:01:44.845 INFO:tasks.ceph.mon.c:Started 2024-09-16T11:01:44.846 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:01:44.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:44.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: get_monmap_and_config 2024-09-16T11:01:44.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: build_initial_monmap 2024-09-16T11:01:44.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: monmap: 2024-09-16T11:01:44.954 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:44.953356+0000 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:44.953356+0000 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:44.955 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 Processor -- start 2024-09-16T11:01:44.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 -- start start 2024-09-16T11:01:44.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: init 2024-09-16T11:01:44.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:44.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 0x7fc2e409f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-c con 0x7fc2e40a1650 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc2e40a0680 0x7fc2e409f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-a con 0x7fc2e40a0680 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2e409cd70 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40a1a30 con 0x7fc2e409cd70 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e409c0b0 con 0x7fc2e40a0680 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e409c390 con 0x7fc2e40a1650 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 10 monclient(hunting): authenticate will time out at 1563.530426s 2024-09-16T11:01:44.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 0x7fc2e409f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc2e40a0680 0x7fc2e409f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 0x7fc2e409f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:49038/0 (socket says 172.21.15.5:49038) 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 1 -- 172.21.15.5:0/531126387 learned_addr learned my addr 172.21.15.5:0/531126387 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc2e40a0680 0x7fc2e409f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54054/0 (socket says 172.21.15.5:54054) 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): get_auth_request con 0x7fc2e40a1650 auth_method 0 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): get_auth_request con 0x7fc2e40a0680 auth_method 0 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:44.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ea24d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_done global_id 74211 payload 274 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/531126387 >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 msgr2=0x7fc2e409f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 --2- 172.21.15.5:0/531126387 >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 0x7fc2e409f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/531126387 >> v1:172.21.15.143:6789/0 conn(0x7fc2e409cd70 legacy=0x7fc2e4138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient: _finish_hunting 0 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 monclient: found mon.noname-a 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:44.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/531126387 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e409c670 con 0x7fc2e40a0680 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e9a4c700 1 --2- 172.21.15.5:0/531126387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e409f800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2e406b9a0 tx=0x7fc2d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d7e845364831de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 1 -- 172.21.15.5:0/531126387 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400e070 con 0x7fc2e40a0680 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 10 monclient: dump: 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:44.969 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 10 monclient: _finish_auth 0 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 10 monclient: _check_auth_tickets 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2e924b700 1 -- 172.21.15.5:0/531126387 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d400bb20 con 0x7fc2e40a0680 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.950+0000 7fc2ec4b1700 5 monclient: authenticate success, global_id 74211 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e924b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:44.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 4 set_mon_vals no callback set 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e924b700 1 -- 172.21.15.5:0/531126387 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4012430 con 0x7fc2e40a0680 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e924b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e924b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e924b700 10 monclient: dump: 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:44.971 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: shutdown 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/531126387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 msgr2=0x7fc2e409f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/531126387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e409f800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2e406b9a0 tx=0x7fc2d400b330 comp rx=0 tx=0).stop 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/531126387 shutdown_connections 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/531126387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e409f800 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/531126387 >> v2:172.21.15.143:3300/0 conn(0x7fc2e40a1650 0x7fc2e409f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/531126387 >> 172.21.15.5:0/531126387 conn(0x7fc2e40a0b30 msgr2=0x7fc2e40a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/531126387 shutdown_connections 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/531126387 wait complete. 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: build_initial_monmap 2024-09-16T11:01:44.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: monmap: 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 librados: starting msgr at 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 librados: starting objecter 2024-09-16T11:01:44.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 Processor -- start 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- start start 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 librados: setting wanted keys 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 librados: calling monclient init 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: init 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 0x7fc2e41cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-b con 0x7fc2e409cd70 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-a con 0x7fc2e40a0680 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): picked mon.noname-c con 0x7fc2e40a1650 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41a7850 con 0x7fc2e40a0680 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41a7b30 con 0x7fc2e409cd70 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e409b1b0 con 0x7fc2e40a1650 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient(hunting): authenticate will time out at 1563.534890s 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54056/0 (socket says 172.21.15.5:54056) 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/567382935 learned_addr learned my addr 172.21.15.5:0/567382935 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): get_auth_request con 0x7fc2e40a0680 auth_method 0 2024-09-16T11:01:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 10 monclient(hunting): get_auth_request con 0x7fc2e40a1650 auth_method 0 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 0x7fc2e41cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 10 monclient(hunting): get_auth_request con 0x7fc2e409cd70 auth_method 0 2024-09-16T11:01:44.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 1 -- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 msgr2=0x7fc2e406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 1 -- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 msgr2=0x7fc2e406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient(hunting): handle_auth_done global_id 74214 payload 824 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2eaa4e700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 msgr2=0x7fc2e406c350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/567382935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 msgr2=0x7fc2e41cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:44.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 0x7fc2e41cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient: _finish_hunting 0 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 monclient: found mon.noname-a 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 -- 172.21.15.5:0/567382935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4009a70 con 0x7fc2e40a0680 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ea24d700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 0x7fc2e41cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e9a4c700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4014040 tx=0x7fc2d4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 client.?.objecter ms_handle_connect 0x7fc2e40a0680 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4012430 con 0x7fc2e40a0680 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 monclient: dump: 2024-09-16T11:01:44.978 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 monclient: _finish_auth 0 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:44.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 5 monclient: authenticate success, global_id 74214 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: _renew_subs 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.954+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e40a3310 con 0x7fc2e40a0680 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4020070 con 0x7fc2e40a0680 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 client.74214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 monclient: _renew_subs 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4004280 con 0x7fc2e40a0680 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e41a7e10 con 0x7fc2e40a0680 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 1 librados: init done 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:44.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e8a4a700 4 set_mon_vals no callback set 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 monclient: dump: 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:44.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc2d40043e0 con 0x7fc2e40a0680 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e41a7570 con 0x7fc2e40a0680 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 1 --2- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 0x7fc2d0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(470..470 src has 344..470) v4 ==== 4875+0+0 (secure 0 0 0) 0x7fc2d4022070 con 0x7fc2e40a0680 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 client.74214.objecter ms_dispatch 0x7fc2e4000fd0 osd_map(470..470 src has 344..470) v4 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 3 client.74214.objecter handle_osd_map got epochs [470,470] > 0 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 3 client.74214.objecter handle_osd_map decoding full epoch 470 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 20 client.74214.objecter dump_active .. 0 homeless 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ea24d700 1 --2- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 0x7fc2d0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:44.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ea24d700 10 monclient: get_auth_request con 0x7fc2d0036f60 auth_method 0 2024-09-16T11:01:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2e2ffd700 10 client.74214.objecter ms_handle_connect 0x7fc2d0036f60 2024-09-16T11:01:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.958+0000 7fc2ea24d700 1 --2- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 0x7fc2d0039420 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009c40 tx=0x7fc2dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.966+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc2d4004920 con 0x7fc2e40a0680 2024-09-16T11:01:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.966+0000 7fc2e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:44.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:44.966+0000 7fc2e2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:45.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2ec4b1700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2ec4b1700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2ec4b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc2e41a6470 con 0x7fc2e40a0680 2024-09-16T11:01:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2e2ffd700 1 -- 172.21.15.5:0/567382935 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fc2d401ab70 con 0x7fc2e40a0680 2024-09-16T11:01:45.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:45.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.202+0000 7fc2e2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:45.209 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":92,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 10 librados: watch_flush enter 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 10 librados: watch_flush exit 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 20 client.74214.objecter shutdown clearing up homeless session... 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 10 client.74214.objecter successfully canceled tick 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 msgr2=0x7fc2d0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:45.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 0x7fc2d0039420 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009c40 tx=0x7fc2dc008040 comp rx=0 tx=0).stop 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 10 monclient: shutdown 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 msgr2=0x7fc2e41cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4014040 tx=0x7fc2d4004af0 comp rx=0 tx=0).stop 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 shutdown_connections 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc2e40a1650 0x7fc2e406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc2d0036f60 0x7fc2d0039420 unknown :-1 s=CLOSED pgs=3838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2e40a0680 0x7fc2e41cf620 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:45.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 --2- 172.21.15.5:0/567382935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2e409cd70 0x7fc2e41cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:45.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 >> 172.21.15.5:0/567382935 conn(0x7fc2e40a0b30 msgr2=0x7fc2e406efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:45.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 shutdown_connections 2024-09-16T11:01:45.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 -- 172.21.15.5:0/567382935 wait complete. 2024-09-16T11:01:45.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:45.206+0000 7fc2ec4b1700 1 librados: shutdown 2024-09-16T11:01:48.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:48.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: get_monmap_and_config 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: build_initial_monmap 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: monmap: 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:48.345589+0000 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:48.345589+0000 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:48.347 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:48.348 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:48.348 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 Processor -- start 2024-09-16T11:01:48.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 -- start start 2024-09-16T11:01:48.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: init 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f39a80a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39a809f780 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39a80a3510 0x7f39a81360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-a con 0x7f39a80a3510 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a80a3c40 con 0x7f39a809f780 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a80a4300 con 0x7f39a80a3510 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:48.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a809fda0 con 0x7f39a80a2bf0 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a759e700 10 monclient(hunting): authenticate will time out at 1566.922222s 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39a80a3510 0x7f39a81360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39a80a3510 0x7f39a81360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54120/0 (socket says 172.21.15.5:54120) 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a5d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f39a809f780 0x7f39a81359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36128/0 (socket says 172.21.15.5:36128) 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 -- 172.21.15.5:0/3016763923 learned_addr learned my addr 172.21.15.5:0/3016763923 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): get_auth_request con 0x7f39a80a3510 auth_method 0 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 10 monclient(hunting): get_auth_request con 0x7f39a80a2bf0 auth_method 0 2024-09-16T11:01:48.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 1 -- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 msgr2=0x7f39a80a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 1 -- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 msgr2=0x7f39a80a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a659c700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient(hunting): handle_auth_done global_id 74223 payload 274 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 -- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 msgr2=0x7f39a80a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 -- 172.21.15.5:0/3016763923 >> v1:172.21.15.5:6789/0 conn(0x7f39a809f780 legacy=0x7f39a81359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:48.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 1 -- 172.21.15.5:0/3016763923 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74141945 0 0) 0x7f39a80a3c40 con 0x7f39a809f780 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 monclient: found mon.noname-a 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 -- 172.21.15.5:0/3016763923 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a80a0080 con 0x7f39a80a3510 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a6d9d700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a80a3510 0x7f39a81360c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009e50 tx=0x7f39a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454aa417398c8152 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 1 -- 172.21.15.5:0/3016763923 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000e070 con 0x7f39a80a3510 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: dump: 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:48.360 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: _finish_auth 0 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 10 monclient: _check_auth_tickets 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.342+0000 7f39a559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 1 -- 172.21.15.5:0/3016763923 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000b9c0 con 0x7f39a80a3510 2024-09-16T11:01:48.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 5 monclient: authenticate success, global_id 74223 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 4 set_mon_vals no callback set 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 1 -- 172.21.15.5:0/3016763923 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000bd70 con 0x7f39a80a3510 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a559a700 10 monclient: dump: 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.362 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: shutdown 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a80a3510 msgr2=0x7f39a81360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a80a3510 0x7f39a81360c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009e50 tx=0x7f39a000b330 comp rx=0 tx=0).stop 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- 172.21.15.5:0/3016763923 shutdown_connections 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a80a3510 0x7f39a81360c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- 172.21.15.5:0/3016763923 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a2bf0 0x7f39a80a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- 172.21.15.5:0/3016763923 >> 172.21.15.5:0/3016763923 conn(0x7f39a80a53e0 msgr2=0x7f39a8073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- 172.21.15.5:0/3016763923 shutdown_connections 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- 172.21.15.5:0/3016763923 wait complete. 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: build_initial_monmap 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: monmap: 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:48.364 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 librados: starting msgr at 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 librados: starting objecter 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 Processor -- start 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- start start 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 librados: setting wanted keys 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 librados: calling monclient init 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: init 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 0x7f39a81ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-a con 0x7f39a809f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39a80a2bf0 0x7f39a81360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-b con 0x7f39a80a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): picked mon.noname-c con 0x7f39a80a3510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a81caae0 con 0x7f39a809f780 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8070430 con 0x7f39a80a2bf0 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a809f4e0 con 0x7f39a80a3510 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a759e700 10 monclient(hunting): authenticate will time out at 1566.927401s 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:50526/0 (socket says 172.21.15.5:50526) 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 -- 172.21.15.5:0/4203706851 learned_addr learned my addr 172.21.15.5:0/4203706851 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 10 monclient(hunting): get_auth_request con 0x7f39a80a3510 auth_method 0 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 0x7f39a81ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): get_auth_request con 0x7f39a809f780 auth_method 0 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 -- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 msgr2=0x7f39a81365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:01:48.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 -- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 msgr2=0x7f39a81365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a6d9d700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a5d9b700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39a80a2bf0 0x7f39a81360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a5d9b700 10 monclient(hunting): get_auth_request con 0x7f39a80a2bf0 auth_method 0 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.346+0000 7f39a5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:48.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 10 monclient(hunting): handle_auth_done global_id 74229 payload 824 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 -- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 msgr2=0x7f39a81365f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 -- 172.21.15.5:0/4203706851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39a80a2bf0 msgr2=0x7f39a81360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39a80a2bf0 0x7f39a81360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 10 monclient: _finish_hunting 0 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 monclient: found mon.noname-a 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 -- 172.21.15.5:0/4203706851 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a0009b30 con 0x7f39a809f780 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a659c700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 0x7f39a81ca5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e7e0 tx=0x7f399c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 client.?.objecter ms_handle_connect 0x7f39a809f780 2024-09-16T11:01:48.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00c070 con 0x7f39a809f780 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: dump: 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:48.370 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: _finish_auth 0 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 5 monclient: authenticate success, global_id 74229 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _renew_subs 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c004020 con 0x7f39a809f780 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a8123a70 con 0x7f39a809f780 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c0043d0 con 0x7f39a809f780 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:48.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a4d99700 4 set_mon_vals no callback set 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 client.74229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 monclient: dump: 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:48.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _renew_subs 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a8123790 con 0x7f39a809f780 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 1 librados: init done 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a80a3920 con 0x7f39a809f780 2024-09-16T11:01:48.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f399c0036c0 con 0x7f39a809f780 2024-09-16T11:01:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 --2- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 0x7f3988039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(474..474 src has 344..474) v4 ==== 5684+0+0 (secure 0 0 0) 0x7f399c047880 con 0x7f39a809f780 2024-09-16T11:01:48.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 10 client.74229.objecter ms_dispatch 0x7f39a8000fd0 osd_map(474..474 src has 344..474) v4 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 3 client.74229.objecter handle_osd_map got epochs [474,474] > 0 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 3 client.74229.objecter handle_osd_map decoding full epoch 474 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f3996ffd700 20 client.74229.objecter dump_active .. 0 homeless 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a5d9b700 1 --2- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 0x7f3988039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.350+0000 7f39a5d9b700 10 monclient: get_auth_request con 0x7f3988036f60 auth_method 0 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.354+0000 7f3996ffd700 10 client.74229.objecter ms_handle_connect 0x7f3988036f60 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.354+0000 7f39a5d9b700 1 --2- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 0x7f3988039420 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7f39a8074040 tx=0x7f3998007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.358+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f39a80a3920 con 0x7f39a809f780 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.358+0000 7f3996ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:48.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.358+0000 7f3996ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:48.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f39a759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:48.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f39a759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:48.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f39a759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:48.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39a80a3920 con 0x7f39a809f780 2024-09-16T11:01:48.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f3996ffd700 1 -- 172.21.15.5:0/4203706851 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f399c017020 con 0x7f39a809f780 2024-09-16T11:01:48.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f3996ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:48.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.594+0000 7f3996ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:48.600 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":95,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:48.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 10 librados: watch_flush enter 2024-09-16T11:01:48.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 10 librados: watch_flush exit 2024-09-16T11:01:48.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 20 client.74229.objecter shutdown clearing up homeless session... 2024-09-16T11:01:48.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 10 client.74229.objecter successfully canceled tick 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 msgr2=0x7f3988039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 0x7f3988039420 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7f39a8074040 tx=0x7f3998007e00 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 10 monclient: shutdown 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 msgr2=0x7f39a81ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 0x7f39a81ca5a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e7e0 tx=0x7f399c00eaf0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 shutdown_connections 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f39a80a3510 0x7f39a81365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39a80a2bf0 0x7f39a81360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3988036f60 0x7f3988039420 unknown :-1 s=CLOSED pgs=3840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 --2- 172.21.15.5:0/4203706851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39a809f780 0x7f39a81ca5a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 >> 172.21.15.5:0/4203706851 conn(0x7f39a80a53e0 msgr2=0x7f39a81bdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:48.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 shutdown_connections 2024-09-16T11:01:48.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 -- 172.21.15.5:0/4203706851 wait complete. 2024-09-16T11:01:48.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:48.598+0000 7f39a759e700 1 librados: shutdown 2024-09-16T11:01:51.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: get_monmap_and_config 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: build_initial_monmap 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: monmap: 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:51.738490+0000 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:51.738490+0000 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:51.740 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 Processor -- start 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 -- start start 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: init 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fada009aa50 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fada0071870 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fada01386f0 0x7fada013aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-c con 0x7fada01386f0 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada009caa0 con 0x7fada0071870 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada009ae30 con 0x7fada009aa50 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada009e3e0 con 0x7fada01386f0 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada5bac700 10 monclient(hunting): authenticate will time out at 1570.314985s 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9ffff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 0x7fada009eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36142/0 (socket says 172.21.15.5:36142) 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9ffff700 1 -- 172.21.15.5:0/3884353851 learned_addr learned my addr 172.21.15.5:0/3884353851 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 1 --2- 172.21.15.5:0/3884353851 >> v2:172.21.15.143:3300/0 conn(0x7fada01386f0 0x7fada013aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): get_auth_request con 0x7fada01386f0 auth_method 0 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:51.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fada53ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114900512 0 0) 0x7fad90001490 con 0x7fada0071870 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 10 monclient(hunting): _negotiate 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 10 monclient(hunting): my global_id is 74232 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad880034f0 con 0x7fada0071870 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.734+0000 7fad9ffff700 1 -- 172.21.15.5:0/3884353851 >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 legacy=0x7fada009eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9ffff700 1 -- 172.21.15.5:0/3884353851 >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 legacy=0x7fada009eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 10 monclient(hunting): handle_auth_done global_id 64336 payload 274 2024-09-16T11:01:51.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 1 -- 172.21.15.5:0/3884353851 >> v1:172.21.15.143:6789/0 conn(0x7fada009aa50 legacy=0x7fada009e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9ffff700 1 --1- 172.21.15.5:0/3884353851 >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 0x7fada009eff0 :-1 s=OPENED pgs=94 cs=1 l=1).handle_message read tag failed 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9ffff700 1 --1- 172.21.15.5:0/3884353851 >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 0x7fada009eff0 :-1 s=OPENED pgs=94 cs=1 l=1).fault on lossy channel, failing 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150582014 0 0) 0x7fada009ae30 con 0x7fada009aa50 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 1 -- 172.21.15.5:0/3884353851 >> v1:172.21.15.5:6789/0 conn(0x7fada0071870 legacy=0x7fada009eff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 10 monclient: _finish_hunting 0 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 1 monclient: found mon.noname-c 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 1 -- 172.21.15.5:0/3884353851 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada009f700 con 0x7fada01386f0 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: ms_handle_reset stray mon v1:172.21.15.5:6789/0 2024-09-16T11:01:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada53ab700 1 --2- 172.21.15.5:0/3884353851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada01386f0 0x7fada013aae0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fada009e520 tx=0x7fad98009ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34ed9988e286fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011830 con 0x7fada01386f0 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: dump: 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:51.748 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 5 monclient: authenticate success, global_id 64336 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad98011990 con 0x7fada01386f0 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 1 -- 172.21.15.5:0/3884353851 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011d00 con 0x7fada01386f0 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 4 set_mon_vals no callback set 2024-09-16T11:01:51.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fad9f7fe700 10 monclient: dump: 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 10 monclient: shutdown 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:51.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 -- 172.21.15.5:0/3884353851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada01386f0 msgr2=0x7fada013aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 --2- 172.21.15.5:0/3884353851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada01386f0 0x7fada013aae0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fada009e520 tx=0x7fad98009ab0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 -- 172.21.15.5:0/3884353851 shutdown_connections 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 --2- 172.21.15.5:0/3884353851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada01386f0 0x7fada013aae0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 -- 172.21.15.5:0/3884353851 >> 172.21.15.5:0/3884353851 conn(0x7fada00a45d0 msgr2=0x7fada00a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 -- 172.21.15.5:0/3884353851 shutdown_connections 2024-09-16T11:01:51.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 1 -- 172.21.15.5:0/3884353851 wait complete. 2024-09-16T11:01:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 10 monclient: build_initial_monmap 2024-09-16T11:01:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.738+0000 7fada5bac700 10 monclient: monmap: 2024-09-16T11:01:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 librados: starting msgr at 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 librados: starting objecter 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 Processor -- start 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- start start 2024-09-16T11:01:51.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 librados: setting wanted keys 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 librados: calling monclient init 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: init 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-a con 0x7fada0071870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada009aa50 0x7fada006f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-b con 0x7fada009aa50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): picked mon.noname-c con 0x7fada01386f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:51.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada0070870 con 0x7fada0071870 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada009b0d0 con 0x7fada009aa50 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada011ba80 con 0x7fada01386f0 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient(hunting): authenticate will time out at 1570.321761s 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54158/0 (socket says 172.21.15.5:54158) 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 -- 172.21.15.5:0/1750490884 learned_addr learned my addr 172.21.15.5:0/1750490884 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): get_auth_request con 0x7fada0071870 auth_method 0 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:51.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 10 monclient(hunting): get_auth_request con 0x7fada01386f0 auth_method 0 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada009aa50 0x7fada006f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 1 -- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 msgr2=0x7fada006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): get_auth_request con 0x7fada009aa50 auth_method 0 2024-09-16T11:01:51.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 1 -- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 msgr2=0x7fada006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada53ab700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient(hunting): handle_auth_done global_id 74235 payload 824 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 -- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 msgr2=0x7fada006f600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 -- 172.21.15.5:0/1750490884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada009aa50 msgr2=0x7fada006f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada009aa50 0x7fada006f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient: _finish_hunting 0 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 monclient: found mon.noname-a 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 -- 172.21.15.5:0/1750490884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980094b0 con 0x7fada0071870 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada4baa700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad9400c520 tx=0x7fad9400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 client.?.objecter ms_handle_connect 0x7fada0071870 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad940054e0 con 0x7fada0071870 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:51.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: dump: 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: _finish_auth 0 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: _check_auth_tickets 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:51.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad940036a0 con 0x7fada0071870 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 5 monclient: authenticate success, global_id 74235 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _renew_subs 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94003a10 con 0x7fada0071870 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada011a630 con 0x7fada0071870 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9effd700 4 set_mon_vals no callback set 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fad9d7fa700 10 monclient: dump: 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:51.764 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 client.74235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _renew_subs 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada011b4c0 con 0x7fada0071870 2024-09-16T11:01:51.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.742+0000 7fada5bac700 1 librados: init done 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fada5bac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fada5bac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fada5bac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada0133070 con 0x7fada0071870 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fad94003b70 con 0x7fada0071870 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 1 --2- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 0x7fad8c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(477..477 src has 344..477) v4 ==== 6084+0+0 (secure 0 0 0) 0x7fad94047520 con 0x7fada0071870 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 10 client.74235.objecter ms_dispatch 0x7fada0000fd0 osd_map(477..477 src has 344..477) v4 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 3 client.74235.objecter handle_osd_map got epochs [477,477] > 0 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 3 client.74235.objecter handle_osd_map decoding full epoch 477 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9ffff700 1 --2- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 0x7fad8c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:51.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 20 client.74235.objecter dump_active .. 0 homeless 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9ffff700 10 monclient: get_auth_request con 0x7fad8c036f60 auth_method 0 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9d7fa700 10 client.74235.objecter ms_handle_connect 0x7fad8c036f60 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.746+0000 7fad9ffff700 1 --2- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 0x7fad8c039420 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7fada011bc90 tx=0x7fad90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.750+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad9401b3b0 con 0x7fada0071870 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.750+0000 7fad9d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:51.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.750+0000 7fad9d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:51.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fada00a10f0 con 0x7fada0071870 2024-09-16T11:01:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fad9d7fa700 1 -- 172.21.15.5:0/1750490884 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fad94017020 con 0x7fada0071870 2024-09-16T11:01:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fad9d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:51.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fad9d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:51.987 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":46,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":99,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 librados: watch_flush enter 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 librados: watch_flush exit 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 20 client.74235.objecter shutdown clearing up homeless session... 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 client.74235.objecter successfully canceled tick 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 msgr2=0x7fad8c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 0x7fad8c039420 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7fada011bc90 tx=0x7fad90009040 comp rx=0 tx=0).stop 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 10 monclient: shutdown 2024-09-16T11:01:51.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 msgr2=0x7fada006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad9400c520 tx=0x7fad9400c830 comp rx=0 tx=0).stop 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 shutdown_connections 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fada01386f0 0x7fada006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fada009aa50 0x7fada006f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad8c036f60 0x7fad8c039420 unknown :-1 s=CLOSED pgs=3841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 --2- 172.21.15.5:0/1750490884 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fada0071870 0x7fada006eb80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 >> 172.21.15.5:0/1750490884 conn(0x7fada00a45d0 msgr2=0x7fada00a26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 shutdown_connections 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 -- 172.21.15.5:0/1750490884 wait complete. 2024-09-16T11:01:51.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:51.982+0000 7fada5bac700 1 librados: shutdown 2024-09-16T11:01:55.002 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: get_monmap_and_config 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: monmap: 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:55.117071+0000 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:55.117071+0000 2024-09-16T11:01:55.118 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:55.119 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 Processor -- start 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 -- start start 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: init 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T11:01:55.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43100a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f431009f780 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43100a36e0 0x7f431013e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-c con 0x7f43100a36e0 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100a3c40 con 0x7f431009f780 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43100a4300 con 0x7f43100a2bf0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431009fda0 con 0x7f43100a36e0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient(hunting): authenticate will time out at 1573.693742s 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430e59c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f431009f780 0x7f431013ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47080/0 (socket says 172.21.15.5:47080) 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430e59c700 1 -- 172.21.15.5:0/1117219855 learned_addr learned my addr 172.21.15.5:0/1117219855 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 --2- 172.21.15.5:0/1117219855 >> v2:172.21.15.143:3300/0 conn(0x7f43100a36e0 0x7f431013e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): get_auth_request con 0x7f43100a36e0 auth_method 0 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 1 -- 172.21.15.5:0/1117219855 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410912170 0 0) 0x7f43100a3c40 con 0x7f431009f780 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient(hunting): _negotiate 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient(hunting): my global_id is 74238 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 1 -- 172.21.15.5:0/1117219855 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc0035d0 con 0x7f431009f780 2024-09-16T11:01:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient(hunting): handle_auth_done global_id 64345 payload 274 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 -- 172.21.15.5:0/1117219855 >> v1:172.21.15.143:6789/0 conn(0x7f43100a2bf0 legacy=0x7f43100a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 -- 172.21.15.5:0/1117219855 >> v1:172.21.15.5:6789/0 conn(0x7f431009f780 legacy=0x7f431013ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 monclient: found mon.noname-c 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 -- 172.21.15.5:0/1117219855 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100a0080 con 0x7f43100a36e0 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430f59e700 1 --2- 172.21.15.5:0/1117219855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43100a36e0 0x7f431013e530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4308009850 tx=0x7f4308009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7227c26957fbec56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 1 -- 172.21.15.5:0/1117219855 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308011830 con 0x7f43100a36e0 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: dump: 2024-09-16T11:01:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 1 -- 172.21.15.5:0/1117219855 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308011990 con 0x7f43100a36e0 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 5 monclient: authenticate success, global_id 64345 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f43148d9700 4 set_mon_vals no callback set 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 1 -- 172.21.15.5:0/1117219855 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308011d00 con 0x7f43100a36e0 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.114+0000 7f430dd9b700 10 monclient: dump: 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.127 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: shutdown 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/1117219855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43100a36e0 msgr2=0x7f431013e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 --2- 172.21.15.5:0/1117219855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43100a36e0 0x7f431013e530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4308009850 tx=0x7f4308009b60 comp rx=0 tx=0).stop 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/1117219855 shutdown_connections 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 --2- 172.21.15.5:0/1117219855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43100a36e0 0x7f431013e530 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/1117219855 >> 172.21.15.5:0/1117219855 conn(0x7f43100a53e0 msgr2=0x7f4310073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/1117219855 shutdown_connections 2024-09-16T11:01:55.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/1117219855 wait complete. 2024-09-16T11:01:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: monmap: 2024-09-16T11:01:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 librados: starting msgr at 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 librados: starting objecter 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 Processor -- start 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- start start 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 librados: setting wanted keys 2024-09-16T11:01:55.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 librados: calling monclient init 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: init 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f431009f780 0x7f43101cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-b con 0x7f431009f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-a con 0x7f43100a2bf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43100a36e0 0x7f431013a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): picked mon.noname-c con 0x7f43100a36e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4310136a80 con 0x7f43100a2bf0 2024-09-16T11:01:55.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43100a7810 con 0x7f431009f780 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431013ced0 con 0x7f43100a36e0 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient(hunting): authenticate will time out at 1573.698420s 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43100a36e0 0x7f431013a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39550/0 (socket says 172.21.15.5:39550) 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 -- 172.21.15.5:0/3575216249 learned_addr learned my addr 172.21.15.5:0/3575216249 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): get_auth_request con 0x7f43100a2bf0 auth_method 0 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): get_auth_request con 0x7f43100a36e0 auth_method 0 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f431009f780 0x7f43101cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): get_auth_request con 0x7f431009f780 auth_method 0 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient(hunting): handle_auth_done global_id 74241 payload 824 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 -- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43100a36e0 msgr2=0x7f431013a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43100a36e0 0x7f431013a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 -- 172.21.15.5:0/3575216249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f431009f780 msgr2=0x7f43101cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f431009f780 0x7f43101cabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 monclient: found mon.noname-a 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 -- 172.21.15.5:0/3575216249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43080094b0 con 0x7f43100a2bf0 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430e59c700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f42f8005930 tx=0x7f42f800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 client.?.objecter ms_handle_connect 0x7f43100a2bf0 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8005d90 con 0x7f43100a2bf0 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: dump: 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.140 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f80070d0 con 0x7f43100a2bf0 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 5 monclient: authenticate success, global_id 74241 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8007480 con 0x7f43100a2bf0 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43077fe700 10 monclient: dump: 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.142 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f430cd99700 4 set_mon_vals no callback set 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: _renew_subs 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.118+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431011f050 con 0x7f43100a2bf0 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 client.74241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 monclient: _renew_subs 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431011ed70 con 0x7f43100a2bf0 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 1 librados: init done 2024-09-16T11:01:55.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f42f80075e0 con 0x7f43100a2bf0 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43100a2fd0 con 0x7f43100a2bf0 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 1 --2- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 0x7f42f0039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(481..481 src has 344..481) v4 ==== 6076+0+0 (secure 0 0 0) 0x7f42f8047490 con 0x7f43100a2bf0 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 10 client.74241.objecter ms_dispatch 0x7f4310000fd0 osd_map(481..481 src has 344..481) v4 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 3 client.74241.objecter handle_osd_map got epochs [481,481] > 0 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 3 client.74241.objecter handle_osd_map decoding full epoch 481 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 20 client.74241.objecter dump_active .. 0 homeless 2024-09-16T11:01:55.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f430ed9d700 1 --2- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 0x7f42f0039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f430ed9d700 10 monclient: get_auth_request con 0x7f42f0036f90 auth_method 0 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f43077fe700 10 client.74241.objecter ms_handle_connect 0x7f42f0036f90 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.122+0000 7f430ed9d700 1 --2- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 0x7f42f0039450 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7f43000012e0 tx=0x7f4300009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.134+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f42f800f260 con 0x7f43100a2bf0 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.134+0000 7f43077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.134+0000 7f43077fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43148d9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43148d9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43148d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f431013cc50 con 0x7f43100a2bf0 2024-09-16T11:01:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43077fe700 1 -- 172.21.15.5:0/3575216249 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f42f8017080 con 0x7f43100a2bf0 2024-09-16T11:01:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:55.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.370+0000 7f43077fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:55.377 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 10 librados: watch_flush enter 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 10 librados: watch_flush exit 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 20 client.74241.objecter shutdown clearing up homeless session... 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 10 client.74241.objecter successfully canceled tick 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 msgr2=0x7f42f0039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 0x7f42f0039450 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7f43000012e0 tx=0x7f4300009040 comp rx=0 tx=0).stop 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 10 monclient: shutdown 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 msgr2=0x7f43101cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f42f8005930 tx=0x7f42f800c9e0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 shutdown_connections 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f43100a36e0 0x7f431013a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> v2:172.21.15.143:6812/1098205034 conn(0x7f42f0036f90 0x7f42f0039450 unknown :-1 s=CLOSED pgs=3842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f43100a2bf0 0x7f43101cb120 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 --2- 172.21.15.5:0/3575216249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f431009f780 0x7f43101cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 >> 172.21.15.5:0/3575216249 conn(0x7f43100a53e0 msgr2=0x7f431006eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 shutdown_connections 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 -- 172.21.15.5:0/3575216249 wait complete. 2024-09-16T11:01:55.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.374+0000 7f43148d9700 1 librados: shutdown 2024-09-16T11:01:55.397 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:01:55.397 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:01:55.397 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:01:55.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.502+0000 7f233075b700 10 monclient: get_monmap_and_config 2024-09-16T11:01:55.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.502+0000 7f233075b700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.502+0000 7f233075b700 10 monclient: monmap: 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:55.508448+0000 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:55.508448+0000 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:55.510 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:55.511 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 Processor -- start 2024-09-16T11:01:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 -- start start 2024-09-16T11:01:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient: init 2024-09-16T11:01:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23280a2eb0 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f23280a4280 0x7f23280a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): picked mon.noname-a con 0x7f23280a4280 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 0x7f232813d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): picked mon.noname-c con 0x7f232813aeb0 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232809fdc0 con 0x7f23280a2eb0 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23280a1e50 con 0x7f23280a4280 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232809fae0 con 0x7f232813aeb0 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient(hunting): authenticate will time out at 1574.085705s 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f23280a4280 0x7f23280a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f23280a4280 0x7f23280a39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39562/0 (socket says 172.21.15.5:39562) 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 1 --2- 172.21.15.5:0/3211975973 >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 0x7f232813d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 -- 172.21.15.5:0/3211975973 learned_addr learned my addr 172.21.15.5:0/3211975973 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): get_auth_request con 0x7f23280a4280 auth_method 0 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): get_auth_request con 0x7f232813aeb0 auth_method 0 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 1 -- 172.21.15.5:0/3211975973 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338159511 0 0) 0x7f232809fdc0 con 0x7f23280a2eb0 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient(hunting): _negotiate 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient(hunting): my global_id is 64357 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 1 -- 172.21.15.5:0/3211975973 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23140035d0 con 0x7f23280a2eb0 2024-09-16T11:01:55.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient(hunting): handle_auth_done global_id 74244 payload 274 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 -- 172.21.15.5:0/3211975973 >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 msgr2=0x7f232813d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 --2- 172.21.15.5:0/3211975973 >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 0x7f232813d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 -- 172.21.15.5:0/3211975973 >> v1:172.21.15.143:6789/0 conn(0x7f23280a2eb0 legacy=0x7f23280a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 monclient: found mon.noname-a 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 -- 172.21.15.5:0/3211975973 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232809efb0 con 0x7f23280a4280 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232ecf8700 1 --2- 172.21.15.5:0/3211975973 >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 0x7f232813d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232dcf6700 1 --2- 172.21.15.5:0/3211975973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a4280 0x7f23280a39a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23280a6360 tx=0x7f2318009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d8524910a47da0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 1 -- 172.21.15.5:0/3211975973 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318011830 con 0x7f23280a4280 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: dump: 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 5 monclient: authenticate success, global_id 74244 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 1 -- 172.21.15.5:0/3211975973 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2318011990 con 0x7f23280a4280 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f233075b700 4 set_mon_vals no callback set 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 1 -- 172.21.15.5:0/3211975973 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318011d20 con 0x7f23280a4280 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.506+0000 7f232d4f5700 10 monclient: dump: 2024-09-16T11:01:55.528 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: shutdown 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/3211975973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a4280 msgr2=0x7f23280a39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- 172.21.15.5:0/3211975973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a4280 0x7f23280a39a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23280a6360 tx=0x7f2318009bf0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/3211975973 shutdown_connections 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- 172.21.15.5:0/3211975973 >> v2:172.21.15.143:3300/0 conn(0x7f232813aeb0 0x7f232813d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- 172.21.15.5:0/3211975973 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a4280 0x7f23280a39a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/3211975973 >> 172.21.15.5:0/3211975973 conn(0x7f23280a4fb0 msgr2=0x7f23280a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/3211975973 shutdown_connections 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/3211975973 wait complete. 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: monmap: 2024-09-16T11:01:55.530 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 librados: starting msgr at 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 librados: starting objecter 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 Processor -- start 2024-09-16T11:01:55.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- start start 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 librados: setting wanted keys 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 librados: calling monclient init 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: init 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f23281ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): picked mon.noname-a con 0x7f23280a2eb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f23281cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): picked mon.noname-b con 0x7f23280a4280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): picked mon.noname-c con 0x7f232813aeb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23280a7250 con 0x7f23280a2eb0 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328120eb0 con 0x7f23280a4280 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232813d510 con 0x7f232813aeb0 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient(hunting): authenticate will time out at 1574.089841s 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f23281ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33434/0 (socket says 172.21.15.5:33434) 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 learned_addr learned my addr 172.21.15.5:0/4019413354 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): get_auth_request con 0x7f232813aeb0 auth_method 0 2024-09-16T11:01:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f23281cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): get_auth_request con 0x7f23280a2eb0 auth_method 0 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 10 monclient(hunting): get_auth_request con 0x7f23280a4280 auth_method 0 2024-09-16T11:01:55.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient(hunting): handle_auth_done global_id 84104 payload 824 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 msgr2=0x7f23281cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f23281cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 msgr2=0x7f23281ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f23281ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 monclient: found mon.noname-c 2024-09-16T11:01:55.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320009a70 con 0x7f232813aeb0 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f23281cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f23281ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f232000efc0 tx=0x7f232000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 client.?.objecter ms_handle_connect 0x7f232813aeb0 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232000cb40 con 0x7f232813aeb0 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: dump: 2024-09-16T11:01:55.536 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232000f040 con 0x7f232813aeb0 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320003a40 con 0x7f232813aeb0 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: dump: 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.538 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 5 monclient: authenticate success, global_id 84104 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ccf4700 4 set_mon_vals no callback set 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _renew_subs 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2318009570 con 0x7f232813aeb0 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 client.84104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _renew_subs 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 msgr2=0x7f23281cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 msgr2=0x7f23281cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f232000efc0 tx=0x7f232000c3a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:55.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f232000efc0 tx=0x7f232000c3a0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2328121470 con 0x7f232813aeb0 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f233075b700 1 librados: init done 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 msgr2=0x7f23281cb180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f232813aeb0 0x7f23281cb180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f230c0072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): picked mon.b con 0x7f23280a4280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 0x7f230c00a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): picked mon.c con 0x7f230c007c50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f230c00cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): picked mon.a con 0x7f23280a2eb0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c00d5d0 con 0x7f23280a2eb0 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c00d710 con 0x7f23280a4280 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c00d850 con 0x7f230c007c50 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f231effd700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.510+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f230c00cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): get_auth_request con 0x7f23280a2eb0 auth_method 0 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 0x7f230c00a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): get_auth_request con 0x7f230c007c50 auth_method 0 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:01:55.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f230c0072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): get_auth_request con 0x7f23280a4280 auth_method 0 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient(hunting): handle_auth_done global_id 84104 payload 293 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 msgr2=0x7f230c0072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f230c0072c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 msgr2=0x7f230c00cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f230c00cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 monclient: found mon.c 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f233075b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f230c00db40 con 0x7f230c007c50 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23280a4280 0x7f230c0072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23280a2eb0 0x7f230c00cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f233075b700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 0x7f23281caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2328067b80 con 0x7f23281ca700 2024-09-16T11:01:55.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 0x7f23281caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient: get_auth_request con 0x7f23281ca700 auth_method 0 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 client.84104.objecter ms_handle_connect 0x7f230c007c50 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232dcf6700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 0x7f230c00a0e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2318016000 tx=0x7f232000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 client.84104.objecter resend_mon_ops 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 10 monclient(hunting): handle_auth_done global_id 74250 payload 824 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23280a5d80 con 0x7f23281ca700 2024-09-16T11:01:55.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318011b00 con 0x7f230c007c50 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: dump: 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.546 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2318011c80 con 0x7f230c007c50 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231ffff700 4 set_mon_vals no callback set 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 client.84104.objecter ms_handle_connect 0x7f23281ca700 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 client.84104.objecter resend_mon_ops 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f231801a450 con 0x7f230c007c50 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 0x7f23281caae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2324001eb0 tx=0x7f2324003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> v2:172.21.15.143:6812/1098205034 conn(0x7f230c03f7f0 0x7f230c041cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.2 v2:172.21.15.5:3301/0 4 ==== osd_map(481..481 src has 344..481) v4 ==== 6076+0+0 (secure 0 0 0) 0x7f23180474c0 con 0x7f230c007c50 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 client.84104.objecter ms_dispatch 0x7f2328000fd0 osd_map(481..481 src has 344..481) v4 2024-09-16T11:01:55.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 3 client.84104.objecter handle_osd_map got epochs [481,481] > 0 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 3 client.84104.objecter handle_osd_map decoding full epoch 481 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> v2:172.21.15.143:6812/1098205034 conn(0x7f230c03f7f0 0x7f230c041cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 20 client.84104.objecter dump_active .. 0 homeless 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23240118d0 con 0x7f23281ca700 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f231effd700 10 monclient: dump: 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.548 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.514+0000 7f232ecf8700 10 monclient: get_auth_request con 0x7f230c03f7f0 auth_method 0 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2324011a30 con 0x7f23281ca700 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 msgr2=0x7f23281caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 0x7f23281caae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2324001eb0 tx=0x7f2324003cd0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 msgr2=0x7f23281caae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281ca700 0x7f23281caae0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2324001eb0 tx=0x7f2324003cd0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f231effd700 10 client.84104.objecter ms_handle_connect 0x7f230c03f7f0 2024-09-16T11:01:55.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.518+0000 7f232ecf8700 1 --2- 172.21.15.5:0/4019413354 >> v2:172.21.15.143:6812/1098205034 conn(0x7f230c03f7f0 0x7f230c041cb0 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f232000c3e0 tx=0x7f2320014000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f233075b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f233075b700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 0x7f2328136630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f233075b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23280a6d50 con 0x7f23281cb020 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 0x7f2328136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient: get_auth_request con 0x7f23281cb020 auth_method 0 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.522+0000 7f232e4f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f232e4f7700 10 monclient(hunting): handle_auth_done global_id 74253 payload 824 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f232e4f7700 1 -- 172.21.15.5:0/4019413354 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f23280a3290 con 0x7f23281cb020 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 client.84104.objecter ms_handle_connect 0x7f23281cb020 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f232e4f7700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 0x7f2328136630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2324001eb0 tx=0x7f2324000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 client.84104.objecter resend_mon_ops 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324024070 con 0x7f23281cb020 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 monclient: dump: 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.553 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f2324015810 con 0x7f23281cb020 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 msgr2=0x7f2328136630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 0x7f2328136630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2324001eb0 tx=0x7f2324000ed0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 msgr2=0x7f2328136630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f231effd700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f23281cb020 0x7f2328136630 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:55.555 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.556 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.557 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.558 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.559 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.560 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.561 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.562 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.563 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.564 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.565 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.566 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:55.567 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:55.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 10 librados: watch_flush enter 2024-09-16T11:01:55.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 10 librados: watch_flush exit 2024-09-16T11:01:55.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 20 client.84104.objecter shutdown clearing up homeless session... 2024-09-16T11:01:55.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 10 client.84104.objecter successfully canceled tick 2024-09-16T11:01:55.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 >> v2:172.21.15.143:6812/1098205034 conn(0x7f230c03f7f0 msgr2=0x7f230c041cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 --2- 172.21.15.5:0/4019413354 >> v2:172.21.15.143:6812/1098205034 conn(0x7f230c03f7f0 0x7f230c041cb0 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f232000c3e0 tx=0x7f2320014000 comp rx=0 tx=0).stop 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 10 monclient: shutdown 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 msgr2=0x7f230c00a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f232ecf8700 1 -- 172.21.15.5:0/4019413354 reap_dead start 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 0x7f230c00a0e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2318016000 tx=0x7f232000b040 comp rx=0 tx=0).stop 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 shutdown_connections 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 --2- 172.21.15.5:0/4019413354 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f230c007c50 0x7f230c00a0e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 >> 172.21.15.5:0/4019413354 conn(0x7f23280a4fb0 msgr2=0x7f232813c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 shutdown_connections 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 -- 172.21.15.5:0/4019413354 wait complete. 2024-09-16T11:01:55.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.526+0000 7f233075b700 1 librados: shutdown 2024-09-16T11:01:55.571 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:55.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:01:55.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:01:55.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.686+0000 7fa13ffff700 10 monclient: get_monmap_and_config 2024-09-16T11:01:55.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.686+0000 7fa13ffff700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.686+0000 7fa13ffff700 10 monclient: monmap: 2024-09-16T11:01:55.693 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:55.692518+0000 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:55.692518+0000 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:55.694 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 Processor -- start 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- start start 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient: init 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T11:01:55.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa14009e8e0 0x7fa14009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fa14009e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fa14009d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa14009dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa140070e40 con 0x7fa14009dfd0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1400a01d0 con 0x7fa14009e8e0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14009f740 con 0x7fa14009d6a0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient(hunting): authenticate will time out at 1574.269000s 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa14009e8e0 0x7fa14009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33464/0 (socket says 172.21.15.5:33464) 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 -- 172.21.15.5:0/1040186926 learned_addr learned my addr 172.21.15.5:0/1040186926 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): get_auth_request con 0x7fa14009d6a0 auth_method 0 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): get_auth_request con 0x7fa14009e8e0 auth_method 0 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 1 -- 172.21.15.5:0/1040186926 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766449549 0 0) 0x7fa140070e40 con 0x7fa14009dfd0 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient(hunting): _negotiate 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient(hunting): my global_id is 64363 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 1 -- 172.21.15.5:0/1040186926 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12c0035d0 con 0x7fa14009dfd0 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient(hunting): handle_auth_done global_id 84107 payload 274 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 -- 172.21.15.5:0/1040186926 >> v2:172.21.15.5:3300/0 conn(0x7fa14009e8e0 msgr2=0x7fa14009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:55.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 --2- 172.21.15.5:0/1040186926 >> v2:172.21.15.5:3300/0 conn(0x7fa14009e8e0 0x7fa14009ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 -- 172.21.15.5:0/1040186926 >> v1:172.21.15.143:6789/0 conn(0x7fa14009dfd0 legacy=0x7fa1401336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 monclient: found mon.noname-b 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 -- 172.21.15.5:0/1040186926 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14009fef0 con 0x7fa14009d6a0 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa1367fc700 1 --2- 172.21.15.5:0/1040186926 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa128009910 tx=0x7fa128009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f539bb5f6e5e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 1 -- 172.21.15.5:0/1040186926 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128011830 con 0x7fa14009d6a0 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: dump: 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.709 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 5 monclient: authenticate success, global_id 84107 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 1 -- 172.21.15.5:0/1040186926 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa128011990 con 0x7fa14009d6a0 2024-09-16T11:01:55.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 4 set_mon_vals no callback set 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 1 -- 172.21.15.5:0/1040186926 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128011d40 con 0x7fa14009d6a0 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13e7fc700 10 monclient: dump: 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.711 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 10 monclient: shutdown 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- 172.21.15.5:0/1040186926 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 msgr2=0x7fa14009f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 --2- 172.21.15.5:0/1040186926 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa128009910 tx=0x7fa128009c20 comp rx=0 tx=0).stop 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- 172.21.15.5:0/1040186926 shutdown_connections 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 --2- 172.21.15.5:0/1040186926 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa14009f200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 --2- 172.21.15.5:0/1040186926 >> v2:172.21.15.5:3300/0 conn(0x7fa14009e8e0 0x7fa14009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- 172.21.15.5:0/1040186926 >> 172.21.15.5:0/1040186926 conn(0x7fa1400a45d0 msgr2=0x7fa1400a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- 172.21.15.5:0/1040186926 shutdown_connections 2024-09-16T11:01:55.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.690+0000 7fa13ffff700 1 -- 172.21.15.5:0/1040186926 wait complete. 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: monmap: 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:55.713 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 librados: starting msgr at 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 librados: starting objecter 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 Processor -- start 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- start start 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 librados: setting wanted keys 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 librados: calling monclient init 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: init 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa1401ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fa14009d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fa14009dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:55.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14009e8e0 0x7fa1401330d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fa14009e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1401ca770 con 0x7fa14009dfd0 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa140122090 con 0x7fa14009e8e0 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa140120c10 con 0x7fa14009d6a0 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): authenticate will time out at 1574.273605s 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39600/0 (socket says 172.21.15.5:39600) 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 -- 172.21.15.5:0/207178956 learned_addr learned my addr 172.21.15.5:0/207178956 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): get_auth_request con 0x7fa14009dfd0 auth_method 0 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa1401ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14009e8e0 0x7fa1401330d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): get_auth_request con 0x7fa14009d6a0 auth_method 0 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): get_auth_request con 0x7fa14009e8e0 auth_method 0 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient(hunting): handle_auth_done global_id 74259 payload 824 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 msgr2=0x7fa1401ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009d6a0 0x7fa1401ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14009e8e0 msgr2=0x7fa1401330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14009e8e0 0x7fa1401330d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 monclient: found mon.noname-a 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128009570 con 0x7fa14009dfd0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa1367fc700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa1400a0310 tx=0x7fa128004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 client.?.objecter ms_handle_connect 0x7fa14009dfd0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128004510 con 0x7fa14009dfd0 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: dump: 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 5 monclient: authenticate success, global_id 74259 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1280036c0 con 0x7fa14009dfd0 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _renew_subs 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa140122570 con 0x7fa14009dfd0 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128003a70 con 0x7fa14009dfd0 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa137fff700 10 monclient: dump: 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 client.74259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _renew_subs 2024-09-16T11:01:55.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa140133690 con 0x7fa14009dfd0 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 librados: init done 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13dffb700 4 set_mon_vals no callback set 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 0x7fa14009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.694+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa140067380 con 0x7fa14009b3e0 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 0x7fa14009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient: get_auth_request con 0x7fa14009b3e0 auth_method 0 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 10 monclient(hunting): handle_auth_done global_id 84113 payload 824 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fa128003bd0 con 0x7fa14009dfd0 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa14009faa0 con 0x7fa14009b3e0 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 0x7fa14009b7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130011d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 0x7fa120039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 client.74259.objecter ms_handle_connect 0x7fa14009b3e0 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 client.74259.objecter resend_mon_ops 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(481..481 src has 344..481) v4 ==== 6076+0+0 (secure 0 0 0) 0x7fa128021030 con 0x7fa14009dfd0 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 client.74259.objecter ms_dispatch 0x7fa140000fd0 osd_map(481..481 src has 344..481) v4 2024-09-16T11:01:55.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 3 client.74259.objecter handle_osd_map got epochs [481,481] > 0 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 3 client.74259.objecter handle_osd_map decoding full epoch 481 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13f7fe700 1 --2- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 0x7fa120039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13f7fe700 10 monclient: get_auth_request con 0x7fa120036f60 auth_method 0 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 20 client.74259.objecter dump_active .. 0 homeless 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa130004240 con 0x7fa14009b3e0 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 monclient: dump: 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fa1300043a0 con 0x7fa14009b3e0 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 msgr2=0x7fa14009b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 0x7fa14009b7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130011d90 comp rx=0 tx=0).stop 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 msgr2=0x7fa14009b7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009b3e0 0x7fa14009b7c0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130011d90 comp rx=0 tx=0).stop 2024-09-16T11:01:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa137fff700 10 client.74259.objecter ms_handle_connect 0x7fa120036f60 2024-09-16T11:01:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.698+0000 7fa13f7fe700 1 --2- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 0x7fa120039420 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7fa138001560 tx=0x7fa138009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1400a3970 con 0x7fa14009bd00 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 10 monclient: get_auth_request con 0x7fa14009bd00 auth_method 0 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.706+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_done global_id 84116 payload 824 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa140134030 con 0x7fa14009bd00 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 client.74259.objecter ms_handle_connect 0x7fa14009bd00 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 client.74259.objecter resend_mon_ops 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 msgr2=0x7fa14006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 msgr2=0x7fa14006d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130004770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa1300119d0 tx=0x7fa130004770 comp rx=0 tx=0).stop 2024-09-16T11:01:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 msgr2=0x7fa14006d2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 0x7fa1200424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 msgr2=0x7fa14006d2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14009bd00 0x7fa14006d2a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa120042b00 con 0x7fa120040100 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 0x7fa1200424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient: get_auth_request con 0x7fa120040100 auth_method 0 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 10 monclient(hunting): handle_auth_done global_id 84119 payload 824 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 -- 172.21.15.5:0/207178956 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa1300043a0 con 0x7fa120040100 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13effd700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 0x7fa1200424f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa130029000 tx=0x7fa130009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 client.74259.objecter ms_handle_connect 0x7fa120040100 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 client.74259.objecter resend_mon_ops 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa130033660 con 0x7fa120040100 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient: dump: 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.732 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7fa1300337c0 con 0x7fa120040100 2024-09-16T11:01:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 msgr2=0x7fa1200424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 0x7fa1200424f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa130029000 tx=0x7fa130009240 comp rx=0 tx=0).stop 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13f7fe700 1 -- 172.21.15.5:0/207178956 reap_dead start 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 msgr2=0x7fa1200424f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa137fff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa120040100 0x7fa1200424f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:55.734 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.735 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.736 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.737 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:55.738 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.739 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:55.740 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.741 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:55.742 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.743 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.744 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.745 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:55.746 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 10 librados: watch_flush enter 2024-09-16T11:01:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 10 librados: watch_flush exit 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 20 client.74259.objecter shutdown clearing up homeless session... 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 10 client.74259.objecter successfully canceled tick 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 msgr2=0x7fa120039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 0x7fa120039420 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7fa138001560 tx=0x7fa138009040 comp rx=0 tx=0).stop 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 10 monclient: shutdown 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 msgr2=0x7fa140132b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa1400a0310 tx=0x7fa128004770 comp rx=0 tx=0).stop 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 shutdown_connections 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa120036f60 0x7fa120039420 unknown :-1 s=CLOSED pgs=3844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 --2- 172.21.15.5:0/207178956 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14009dfd0 0x7fa140132b90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 >> 172.21.15.5:0/207178956 conn(0x7fa1400a45d0 msgr2=0x7fa14009ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 shutdown_connections 2024-09-16T11:01:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 -- 172.21.15.5:0/207178956 wait complete. 2024-09-16T11:01:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.710+0000 7fa13ffff700 1 librados: shutdown 2024-09-16T11:01:55.749 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:55.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:01:55.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:01:55.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: get_monmap_and_config 2024-09-16T11:01:55.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: monmap: 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:55.874626+0000 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:55.874626+0000 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:55.876 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 Processor -- start 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 -- start start 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: init 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdec40a0db0 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-b con 0x7fdec40a2bd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdec4070940 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40a73e0 con 0x7fdec40a0db0 2024-09-16T11:01:55.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40a2540 con 0x7fdec4070940 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4070240 con 0x7fdec40a2bd0 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdeca4ee700 10 monclient(hunting): authenticate will time out at 1574.451222s 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec3fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdec40a0db0 0x7fdec40a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47088/0 (socket says 172.21.15.5:47088) 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33504/0 (socket says 172.21.15.5:33504) 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec3fff700 1 -- 172.21.15.5:0/3529654490 learned_addr learned my addr 172.21.15.5:0/3529654490 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): get_auth_request con 0x7fdec40a2bd0 auth_method 0 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.870+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 10 monclient(hunting): handle_auth_done global_id 84122 payload 274 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 1 -- 172.21.15.5:0/3529654490 >> v1:172.21.15.143:6789/0 conn(0x7fdec4070940 legacy=0x7fdec413b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 1 -- 172.21.15.5:0/3529654490 >> v1:172.21.15.5:6789/0 conn(0x7fdec40a0db0 legacy=0x7fdec40a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 1 monclient: found mon.noname-b 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 1 -- 172.21.15.5:0/3529654490 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec406fc80 con 0x7fdec40a2bd0 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec37fe700 1 --2- 172.21.15.5:0/3529654490 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009dc0 tx=0x7fdeac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abafccbb9697ddff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 1 -- 172.21.15.5:0/3529654490 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac00e070 con 0x7fdec40a2bd0 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: dump: 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 5 monclient: authenticate success, global_id 84122 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 1 -- 172.21.15.5:0/3529654490 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac00bbd0 con 0x7fdec40a2bd0 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 4 set_mon_vals no callback set 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 1 -- 172.21.15.5:0/3529654490 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac012430 con 0x7fdec40a2bd0 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdec2ffd700 10 monclient: dump: 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.884 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 10 monclient: shutdown 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- 172.21.15.5:0/3529654490 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 msgr2=0x7fdec4138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/3529654490 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009dc0 tx=0x7fdeac00b330 comp rx=0 tx=0).stop 2024-09-16T11:01:55.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- 172.21.15.5:0/3529654490 shutdown_connections 2024-09-16T11:01:55.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/3529654490 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a2bd0 0x7fdec4138eb0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- 172.21.15.5:0/3529654490 >> 172.21.15.5:0/3529654490 conn(0x7fdec40a64b0 msgr2=0x7fdec40a68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- 172.21.15.5:0/3529654490 shutdown_connections 2024-09-16T11:01:55.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- 172.21.15.5:0/3529654490 wait complete. 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 10 monclient: build_initial_monmap 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 10 monclient: monmap: 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 librados: starting msgr at 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 librados: starting objecter 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 Processor -- start 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 -- start start 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 librados: setting wanted keys 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 1 librados: calling monclient init 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.874+0000 7fdeca4ee700 10 monclient: init 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec4070940 0x7fdec41c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-b con 0x7fdec4070940 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a0db0 0x7fdec41c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-c con 0x7fdec40a0db0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 0x7fdec41c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): picked mon.noname-a con 0x7fdec40a2bd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec41a9e70 con 0x7fdec40a2bd0 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec409e140 con 0x7fdec4070940 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec409d6a0 con 0x7fdec40a0db0 2024-09-16T11:01:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): authenticate will time out at 1574.456591s 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 0x7fdec41c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a0db0 0x7fdec41c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 0x7fdec41c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39612/0 (socket says 172.21.15.5:39612) 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec4070940 0x7fdec41c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 learned_addr learned my addr 172.21.15.5:0/2429178526 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): get_auth_request con 0x7fdec40a2bd0 auth_method 0 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): get_auth_request con 0x7fdec40a0db0 auth_method 0 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): get_auth_request con 0x7fdec4070940 auth_method 0 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient(hunting): handle_auth_done global_id 74265 payload 824 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a0db0 msgr2=0x7fdec41c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a0db0 0x7fdec41c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec4070940 msgr2=0x7fdec41c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec4070940 0x7fdec41c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient: _finish_hunting 0 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 monclient: found mon.noname-a 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac009a70 con 0x7fdec40a2bd0 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec4070940 0x7fdec41c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec37fe700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdec40a0db0 0x7fdec41c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec8a8b700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 0x7fdec41c8d60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00c480 tx=0x7fdebc00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 client.?.objecter ms_handle_connect 0x7fdec40a2bd0 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0052d0 con 0x7fdec40a2bd0 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: dump: 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: _finish_auth 0 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 5 monclient: authenticate success, global_id 74265 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc005430 con 0x7fdec40a2bd0 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0055a0 con 0x7fdec40a2bd0 2024-09-16T11:01:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _renew_subs 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec27fc700 4 set_mon_vals no callback set 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec41a7c30 con 0x7fdec40a2bd0 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec0ff9700 10 monclient: dump: 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 client.74265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _renew_subs 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec41aa710 con 0x7fdec40a2bd0 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 librados: init done 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 0x7fdec41c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40677f0 con 0x7fdec41c92a0 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 0x7fdec41c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient: get_auth_request con 0x7fdec41c92a0 auth_method 0 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.878+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fdebc0038e0 con 0x7fdec40a2bd0 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec3fff700 10 monclient(hunting): handle_auth_done global_id 64375 payload 824 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec3fff700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdec406fff0 con 0x7fdec41c92a0 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 --2- 172.21.15.5:0/2429178526 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdea4036f60 0x7fdea4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(481..481 src has 344..481) v4 ==== 6076+0+0 (secure 0 0 0) 0x7fdebc047430 con 0x7fdec40a2bd0 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 client.74265.objecter ms_dispatch 0x7fdec4000fd0 osd_map(481..481 src has 344..481) v4 2024-09-16T11:01:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 3 client.74265.objecter handle_osd_map got epochs [481,481] > 0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 3 client.74265.objecter handle_osd_map decoding full epoch 481 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec3fff700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 0x7fdec41c9680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4007650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 20 client.74265.objecter dump_active .. 0 homeless 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 client.74265.objecter ms_handle_connect 0x7fdec41c92a0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 client.74265.objecter resend_mon_ops 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec37fe700 1 --2- 172.21.15.5:0/2429178526 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdea4036f60 0x7fdea4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec37fe700 10 monclient: get_auth_request con 0x7fdea4036f60 auth_method 0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4004a40 con 0x7fdec41c92a0 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 monclient: dump: 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec37fe700 1 --2- 172.21.15.5:0/2429178526 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdea4036f60 0x7fdea4039420 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b370 tx=0x7fdeac011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 client.74265.objecter ms_handle_connect 0x7fdea4036f60 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fdeb400f850 con 0x7fdec41c92a0 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 msgr2=0x7fdec41c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 0x7fdec41c9680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4007650 comp rx=0 tx=0).stop 2024-09-16T11:01:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 msgr2=0x7fdec41c9680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.882+0000 7fdec0ff9700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41c92a0 0x7fdec41c9680 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4007650 comp rx=0 tx=0).stop 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdeca4ee700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 0x7fdec41a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdeca4ee700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec41c9bc0 con 0x7fdec41a2820 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 0x7fdec41a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient: get_auth_request con 0x7fdec41a2820 auth_method 0 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 10 monclient(hunting): handle_auth_done global_id 64378 payload 824 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 1 -- 172.21.15.5:0/2429178526 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdec40a3e10 con 0x7fdec41a2820 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 10 client.74265.objecter ms_handle_connect 0x7fdec41a2820 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec3fff700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 0x7fdec41a2c00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 10 client.74265.objecter resend_mon_ops 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4023070 con 0x7fdec41a2820 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.894+0000 7fdec0ff9700 10 monclient: dump: 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7fdeb4013600 con 0x7fdec41a2820 2024-09-16T11:01:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 msgr2=0x7fdec41a2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 0x7fdec41a2c00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4000ed0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 msgr2=0x7fdec41a2c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec0ff9700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdec41a2820 0x7fdec41a2c00 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007290 tx=0x7fdeb4000ed0 comp rx=0 tx=0).stop 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:01:55.909 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.910 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.911 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:55.912 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:55.913 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:55.914 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:55.915 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.916 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:55.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:55.919 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.920 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.921 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:55.922 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:55.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 10 librados: watch_flush enter 2024-09-16T11:01:55.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 10 librados: watch_flush exit 2024-09-16T11:01:55.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 20 client.74265.objecter shutdown clearing up homeless session... 2024-09-16T11:01:55.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 10 client.74265.objecter successfully canceled tick 2024-09-16T11:01:55.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdea4036f60 msgr2=0x7fdea4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/2429178526 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdea4036f60 0x7fdea4039420 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00b370 tx=0x7fdeac011000 comp rx=0 tx=0).stop 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 10 monclient: shutdown 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 msgr2=0x7fdec41c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 --2- 172.21.15.5:0/2429178526 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdec40a2bd0 0x7fdec41c8d60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00c480 tx=0x7fdebc00c790 comp rx=0 tx=0).stop 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 reap_dead start 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdec8a8b700 1 -- 172.21.15.5:0/2429178526 reap_dead start 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 shutdown_connections 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 >> 172.21.15.5:0/2429178526 conn(0x7fdec40a64b0 msgr2=0x7fdec40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 shutdown_connections 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 -- 172.21.15.5:0/2429178526 wait complete. 2024-09-16T11:01:55.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:55.898+0000 7fdeca4ee700 1 librados: shutdown 2024-09-16T11:01:55.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:55.926 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T11:01:55.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: get_monmap_and_config 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: build_initial_monmap 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: monmap: 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:56.046369+0000 2024-09-16T11:01:56.047 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:56.046369+0000 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 Processor -- start 2024-09-16T11:01:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 -- start start 2024-09-16T11:01:56.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: init 2024-09-16T11:01:56.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-a con 0x7f1ae00a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ae009f780 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ae00a3510 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00a3c40 con 0x7f1ae009f780 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae01367d0 con 0x7f1ae00a3510 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae00a4300 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:56.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.042+0000 7f1ae49ae700 10 monclient(hunting): authenticate will time out at 1574.623850s 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39624/0 (socket says 172.21.15.5:39624) 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2115964137 learned_addr learned my addr 172.21.15.5:0/2115964137 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): get_auth_request con 0x7f1ae00a2bf0 auth_method 0 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adeffd700 1 --1- 172.21.15.5:0/2115964137 >> v1:172.21.15.5:6789/0 conn(0x7f1ae009f780 0x7f1ae01359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47100/0 (socket says 172.21.15.5:47100) 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:56.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_done global_id 74268 payload 274 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2115964137 >> v1:172.21.15.143:6789/0 conn(0x7f1ae00a3510 legacy=0x7f1ae01360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2115964137 >> v1:172.21.15.5:6789/0 conn(0x7f1ae009f780 legacy=0x7f1ae01359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 monclient: found mon.noname-a 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2115964137 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae009fda0 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1adf7fe700 1 --2- 172.21.15.5:0/2115964137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009910 tx=0x7f1ad0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f609726cf44ca904 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 1 -- 172.21.15.5:0/2115964137 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000ba60 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:56.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: dump: 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: _finish_auth 0 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:01:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 5 monclient: authenticate success, global_id 74268 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 1 -- 172.21.15.5:0/2115964137 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad000bbc0 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 1 -- 172.21.15.5:0/2115964137 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0009e60 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 4 set_mon_vals no callback set 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ade7fc700 10 monclient: dump: 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.055 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:56.056 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 10 monclient: shutdown 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2115964137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a2bf0 msgr2=0x7f1ae00a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2115964137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009910 tx=0x7f1ad0009c20 comp rx=0 tx=0).stop 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2115964137 shutdown_connections 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2115964137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae00a2fd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2115964137 >> 172.21.15.5:0/2115964137 conn(0x7f1ae00a53e0 msgr2=0x7f1ae0073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2115964137 shutdown_connections 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2115964137 wait complete. 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 10 monclient: build_initial_monmap 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 10 monclient: monmap: 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:56.061 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 librados: starting msgr at 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 librados: starting objecter 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.046+0000 7f1ae49ae700 1 Processor -- start 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- start start 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 librados: setting wanted keys 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 librados: calling monclient init 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: init 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-c con 0x7f1ae009f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:56.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae01cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-b con 0x7f1ae00a2bf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a3510 0x7f1ae008b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): picked mon.noname-a con 0x7f1ae00a3510 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae008ec80 con 0x7f1ae00a3510 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae008e9a0 con 0x7f1ae00a2bf0 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae009f4e0 con 0x7f1ae009f780 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient(hunting): authenticate will time out at 1574.628713s 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33520/0 (socket says 172.21.15.5:33520) 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2628959064 learned_addr learned my addr 172.21.15.5:0/2628959064 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): get_auth_request con 0x7f1ae009f780 auth_method 0 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient(hunting): handle_auth_done global_id 84128 payload 824 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2628959064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ae00a2bf0 msgr2=0x7f1ae01cf3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adeffd700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae01cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae01cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a3510 msgr2=0x7f1ae008b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a3510 0x7f1ae008b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 monclient: found mon.noname-c 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 -- 172.21.15.5:0/2628959064 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009570 con 0x7f1ae009f780 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1adf7fe700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 client.?.objecter ms_handle_connect 0x7f1ae009f780 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00036c0 con 0x7f1ae009f780 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:56.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: dump: 2024-09-16T11:01:56.067 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: _finish_auth 0 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: _check_auth_tickets 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:56.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 5 monclient: authenticate success, global_id 84128 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _renew_subs 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad001f070 con 0x7f1ae009f780 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae008e6c0 con 0x7f1ae009f780 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0003a60 con 0x7f1ae009f780 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ac7fff700 10 monclient: dump: 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:56.069 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 client.84128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _renew_subs 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae01cf910 con 0x7f1ae009f780 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1ae49ae700 1 librados: init done 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.050+0000 7f1addffb700 4 set_mon_vals no callback set 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1ad0003bc0 con 0x7f1ae009f780 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ae49ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:56.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ae49ae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ae49ae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0005100 con 0x7f1ae009f780 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 1 --2- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 0x7f1acc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(481..481 src has 344..481) v4 ==== 6076+0+0 (secure 0 0 0) 0x7f1ad0021070 con 0x7f1ae009f780 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 10 client.84128.objecter ms_dispatch 0x7f1ae0000fd0 osd_map(481..481 src has 344..481) v4 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 3 client.84128.objecter handle_osd_map got epochs [481,481] > 0 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 3 client.84128.objecter handle_osd_map decoding full epoch 481 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 20 client.84128.objecter dump_active .. 0 homeless 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1adeffd700 1 --2- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 0x7f1acc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1adeffd700 10 monclient: get_auth_request con 0x7f1acc036f60 auth_method 0 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1ac7fff700 10 client.84128.objecter ms_handle_connect 0x7f1acc036f60 2024-09-16T11:01:56.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.054+0000 7f1adeffd700 1 --2- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 0x7f1acc039420 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000f60 tx=0x7f1ad4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:56.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.062+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1ad0030b50 con 0x7f1ae009f780 2024-09-16T11:01:56.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.062+0000 7f1ac7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:56.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.062+0000 7f1ac7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ae49ae700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T11:01:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ae49ae700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T11:01:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ae49ae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:56.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1ac0005bc0 con 0x7f1ae009f780 2024-09-16T11:01:56.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ac7fff700 1 -- 172.21.15.5:0/2628959064 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1ad002b030 con 0x7f1ae009f780 2024-09-16T11:01:56.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ac7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T11:01:56.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f1ac7fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=2061032040,config=692619927,health=2188991307,kv=564032378,logm=3711879854}) 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=1861093969,config=692619927,health=2188991307,kv=564032378,logm=912809159}) 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.294+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=2061032040,config=692619927,health=2188991307,kv=564032378,logm=3711879854}) 2024-09-16T11:01:56.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=11,kv=10,logm=62} crc {auth=2661449103,config=692619927,health=719047087,kv=564032378,logm=2432610771}) 2024-09-16T11:01:56.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228070194}) 2024-09-16T11:01:56.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612871198}) 2024-09-16T11:01:56.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228070194}) 2024-09-16T11:01:56.305 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.298+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358735255}) 2024-09-16T11:01:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 10 librados: watch_flush enter 2024-09-16T11:01:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 10 librados: watch_flush exit 2024-09-16T11:01:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 20 client.84128.objecter shutdown clearing up homeless session... 2024-09-16T11:01:56.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 10 client.84128.objecter successfully canceled tick 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 msgr2=0x7f1acc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 0x7f1acc039420 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000f60 tx=0x7f1ad4009040 comp rx=0 tx=0).stop 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 10 monclient: shutdown 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 msgr2=0x7f1ae01cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad0004770 comp rx=0 tx=0).stop 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 shutdown_connections 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1acc036f60 0x7f1acc039420 unknown :-1 s=CLOSED pgs=3846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ae00a3510 0x7f1ae008b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ae00a2bf0 0x7f1ae01cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 --2- 172.21.15.5:0/2628959064 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1ae009f780 0x7f1ae01cee90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:56.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 >> 172.21.15.5:0/2628959064 conn(0x7f1ae00a53e0 msgr2=0x7f1ae0070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:56.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 shutdown_connections 2024-09-16T11:01:56.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 -- 172.21.15.5:0/2628959064 wait complete. 2024-09-16T11:01:56.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f1ae49ae700 1 librados: shutdown 2024-09-16T11:01:56.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432234491}) 2024-09-16T11:01:56.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3103219996}) 2024-09-16T11:01:56.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432234491}) 2024-09-16T11:01:56.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.302+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1692942009}) 2024-09-16T11:01:56.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061440666}) 2024-09-16T11:01:56.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1700897887}) 2024-09-16T11:01:56.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061440666}) 2024-09-16T11:01:56.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794285384}) 2024-09-16T11:01:56.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332819495}) 2024-09-16T11:01:56.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809992486}) 2024-09-16T11:01:56.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332819495}) 2024-09-16T11:01:56.314 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.306+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061096567}) 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305637228}) 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949937986}) 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305637228}) 2024-09-16T11:01:56.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063198830}) 2024-09-16T11:01:56.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355078461}) 2024-09-16T11:01:56.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011311612}) 2024-09-16T11:01:56.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355078461}) 2024-09-16T11:01:56.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.310+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063771642}) 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488157459}) 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993110077}) 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488157459}) 2024-09-16T11:01:56.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562313667}) 2024-09-16T11:01:56.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335427420}) 2024-09-16T11:01:56.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3829327973}) 2024-09-16T11:01:56.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335427420}) 2024-09-16T11:01:56.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482132391}) 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320132846}) 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806447155}) 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320132846}) 2024-09-16T11:01:56.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.314+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258335209}) 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500657655}) 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193196504}) 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500657655}) 2024-09-16T11:01:56.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1223952499}) 2024-09-16T11:01:56.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T11:01:56.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222054794}) 2024-09-16T11:01:56.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840586670}) 2024-09-16T11:01:56.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222054794}) 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733903731}) 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821088024}) 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403101336}) 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821088024}) 2024-09-16T11:01:56.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.318+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832822996}) 2024-09-16T11:01:56.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190287940}) 2024-09-16T11:01:56.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056478589}) 2024-09-16T11:01:56.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190287940}) 2024-09-16T11:01:56.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276028959}) 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407926944}) 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971694305}) 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407926944}) 2024-09-16T11:01:56.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235556174}) 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440411953}) 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4019628393}) 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440411953}) 2024-09-16T11:01:56.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.322+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933659659}) 2024-09-16T11:01:56.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171135}) 2024-09-16T11:01:56.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4132616731}) 2024-09-16T11:01:56.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171135}) 2024-09-16T11:01:56.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573624843}) 2024-09-16T11:01:56.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466728121}) 2024-09-16T11:01:56.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054712209}) 2024-09-16T11:01:56.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466728121}) 2024-09-16T11:01:56.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130996899}) 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811943710}) 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726216482}) 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811943710}) 2024-09-16T11:01:56.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.326+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826154258}) 2024-09-16T11:01:56.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542485485}) 2024-09-16T11:01:56.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699492667}) 2024-09-16T11:01:56.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542485485}) 2024-09-16T11:01:56.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601318771}) 2024-09-16T11:01:56.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080369109}) 2024-09-16T11:01:56.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251420978}) 2024-09-16T11:01:56.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3080369109}) 2024-09-16T11:01:56.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395188852}) 2024-09-16T11:01:56.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817315477}) 2024-09-16T11:01:56.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847817069}) 2024-09-16T11:01:56.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817315477}) 2024-09-16T11:01:56.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3591684783}) 2024-09-16T11:01:56.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801120628}) 2024-09-16T11:01:56.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1652371817}) 2024-09-16T11:01:56.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801120628}) 2024-09-16T11:01:56.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.330+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3247671934}) 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642632275}) 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=158119183}) 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642632275}) 2024-09-16T11:01:56.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014022166}) 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149309380}) 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86678816}) 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149309380}) 2024-09-16T11:01:56.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171627002}) 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219072265}) 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861742358}) 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219072265}) 2024-09-16T11:01:56.341 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.334+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107411357}) 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687232129}) 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083148336}) 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687232129}) 2024-09-16T11:01:56.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2809328782}) 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402060141}) 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1258409802}) 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402060141}) 2024-09-16T11:01:56.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.338+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99029974}) 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860842367}) 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3192543683}) 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860842367}) 2024-09-16T11:01:56.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2086341552}) 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299557675}) 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402882414}) 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299557675}) 2024-09-16T11:01:56.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612416833}) 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446890449}) 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=358531101}) 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446890449}) 2024-09-16T11:01:56.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.342+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1957535387}) 2024-09-16T11:01:56.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825343032}) 2024-09-16T11:01:56.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60733876}) 2024-09-16T11:01:56.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825343032}) 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067535}) 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281441240}) 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497228250}) 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281441240}) 2024-09-16T11:01:56.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1642639805}) 2024-09-16T11:01:56.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027505459}) 2024-09-16T11:01:56.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4120532750}) 2024-09-16T11:01:56.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027505459}) 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=336903955}) 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980667452}) 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332942489}) 2024-09-16T11:01:56.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980667452}) 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.346+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46589774}) 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190716487}) 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764962585}) 2024-09-16T11:01:56.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190716487}) 2024-09-16T11:01:56.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2001260309}) 2024-09-16T11:01:56.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071997136}) 2024-09-16T11:01:56.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3761266852}) 2024-09-16T11:01:56.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071997136}) 2024-09-16T11:01:56.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1104059911}) 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658970859}) 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3869365287}) 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658970859}) 2024-09-16T11:01:56.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.350+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129558456}) 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470947278}) 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748915759}) 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470947278}) 2024-09-16T11:01:56.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484794363}) 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775754269}) 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3780085773}) 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775754269}) 2024-09-16T11:01:56.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264052491}) 2024-09-16T11:01:56.360 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.360 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476728710}) 2024-09-16T11:01:56.360 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345568811}) 2024-09-16T11:01:56.360 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.360 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476728710}) 2024-09-16T11:01:56.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739658912}) 2024-09-16T11:01:56.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264863799}) 2024-09-16T11:01:56.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639513626}) 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264863799}) 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.354+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751222290}) 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011215612}) 2024-09-16T11:01:56.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771285907}) 2024-09-16T11:01:56.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011215612}) 2024-09-16T11:01:56.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532022329}) 2024-09-16T11:01:56.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070100162}) 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204878066}) 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1070100162}) 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358324367}) 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221537356}) 2024-09-16T11:01:56.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4124204336}) 2024-09-16T11:01:56.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221537356}) 2024-09-16T11:01:56.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460782448}) 2024-09-16T11:01:56.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771591755}) 2024-09-16T11:01:56.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2653439326}) 2024-09-16T11:01:56.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771591755}) 2024-09-16T11:01:56.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.358+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593381183}) 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=2,mgr=5} crc {logm=2401999161,mdsmap=3585761405,mgr=1836277893}) 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=2391326776,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419}) 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=2,mgr=5} crc {logm=2401999161,mdsmap=3585761405,mgr=1836277893}) 2024-09-16T11:01:56.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1758652192,mdsmap=1531151339,mgr=2661550515}) 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=44,osdmap=22} crc {mgr=543305786,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=680553620,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_snap=3791655142,osdmap=360236367}) 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=22} crc {mgr=543305786,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=462665919,mon_config_key=3273021363,monmap=2357222013,osd_metadata=348837743,osd_pg_creating=1865909500,osd_snap=841288704,osdmap=1954542841}) 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=44,osdmap=22} crc {mgr=543305786,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=680553620,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_snap=3791655142,osdmap=360236367}) 2024-09-16T11:01:56.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=22} crc {mgr=543305786,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=680553620,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=1865909500,osd_snap=3271965814,osdmap=482904986}) 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4041211504}) 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2348431854}) 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4041211504}) 2024-09-16T11:01:56.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.362+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=576378952}) 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=92} crc {osdmap=1718456099}) 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=2077264901}) 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=92} crc {osdmap=1718456099}) 2024-09-16T11:01:56.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:01:56.370+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=89} crc {osdmap=1564080207}) 2024-09-16T11:01:57.324 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:01:57.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.434+0000 7f4797fff700 10 monclient: get_monmap_and_config 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.434+0000 7f4797fff700 10 monclient: build_initial_monmap 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.434+0000 7f4797fff700 10 monclient: monmap: 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:57.440218+0000 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:57.440218+0000 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:57.442 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 Processor -- start 2024-09-16T11:01:57.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- start start 2024-09-16T11:01:57.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient: init 2024-09-16T11:01:57.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47980a2c80 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47980a3770 0x7f479813ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-c con 0x7f47980a3770 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f479813b1c0 0x7f479813d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-a con 0x7f479813b1c0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980a3db0 con 0x7f47980a2c80 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47980a4b70 con 0x7f479813b1c0 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47980a0850 con 0x7f47980a3770 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient(hunting): authenticate will time out at 1576.017206s 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f479813b1c0 0x7f479813d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f479813b1c0 0x7f479813d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39642/0 (socket says 172.21.15.5:39642) 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 -- 172.21.15.5:0/1003265067 learned_addr learned my addr 172.21.15.5:0/1003265067 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): get_auth_request con 0x7f479813b1c0 auth_method 0 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47967fc700 1 --2- 172.21.15.5:0/1003265067 >> v2:172.21.15.143:3300/0 conn(0x7f47980a3770 0x7f479813ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47967fc700 10 monclient(hunting): get_auth_request con 0x7f47980a3770 auth_method 0 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient(hunting): handle_auth_done global_id 74277 payload 274 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 -- 172.21.15.5:0/1003265067 >> v2:172.21.15.143:3300/0 conn(0x7f47980a3770 msgr2=0x7f479813ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 --2- 172.21.15.5:0/1003265067 >> v2:172.21.15.143:3300/0 conn(0x7f47980a3770 0x7f479813ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 -- 172.21.15.5:0/1003265067 >> v1:172.21.15.5:6789/0 conn(0x7f47980a2c80 legacy=0x7f47980a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient: _finish_hunting 0 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 monclient: found mon.noname-a 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:57.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 -- 172.21.15.5:0/1003265067 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980a4180 con 0x7f479813b1c0 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f47977fe700 1 --2- 172.21.15.5:0/1003265067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f479813b1c0 0x7f479813d5b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4798072b50 tx=0x7f4790009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85aba61979e088 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 1 -- 172.21.15.5:0/1003265067 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479000b8a0 con 0x7f479813b1c0 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: dump: 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.449 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: _finish_auth 0 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 5 monclient: authenticate success, global_id 74277 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 1 -- 172.21.15.5:0/1003265067 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479000ba00 con 0x7f479813b1c0 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:57.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 4 set_mon_vals no callback set 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 1 -- 172.21.15.5:0/1003265067 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479000bd90 con 0x7f479813b1c0 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4795ffb700 10 monclient: dump: 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.451 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 10 monclient: shutdown 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- 172.21.15.5:0/1003265067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f479813b1c0 msgr2=0x7f479813d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 --2- 172.21.15.5:0/1003265067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f479813b1c0 0x7f479813d5b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4798072b50 tx=0x7f4790009bf0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- 172.21.15.5:0/1003265067 shutdown_connections 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 --2- 172.21.15.5:0/1003265067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f479813b1c0 0x7f479813d5b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 --2- 172.21.15.5:0/1003265067 >> v2:172.21.15.143:3300/0 conn(0x7f47980a3770 0x7f479813ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- 172.21.15.5:0/1003265067 >> 172.21.15.5:0/1003265067 conn(0x7f47980a4d80 msgr2=0x7f47980a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- 172.21.15.5:0/1003265067 shutdown_connections 2024-09-16T11:01:57.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.438+0000 7f4797fff700 1 -- 172.21.15.5:0/1003265067 wait complete. 2024-09-16T11:01:57.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient: build_initial_monmap 2024-09-16T11:01:57.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient: monmap: 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 librados: starting msgr at 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 librados: starting objecter 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 Processor -- start 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 -- start start 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 librados: setting wanted keys 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 librados: calling monclient init 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient: init 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47980a2c80 0x7f4798070520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-b con 0x7f47980a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-a con 0x7f47980a3770 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f479813b1c0 0x7f4798135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): picked mon.noname-c con 0x7f479813b1c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479809f4e0 con 0x7f47980a3770 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4798071110 con 0x7f47980a2c80 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4798120ed0 con 0x7f479813b1c0 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f4797fff700 10 monclient(hunting): authenticate will time out at 1576.023852s 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.442+0000 7f47967fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39658/0 (socket says 172.21.15.5:39658) 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f479813b1c0 0x7f4798135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 -- 172.21.15.5:0/900933787 learned_addr learned my addr 172.21.15.5:0/900933787 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): get_auth_request con 0x7f47980a3770 auth_method 0 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47980a2c80 0x7f4798070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): get_auth_request con 0x7f47980a2c80 auth_method 0 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): get_auth_request con 0x7f479813b1c0 auth_method 0 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient(hunting): handle_auth_done global_id 74283 payload 824 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 -- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f479813b1c0 msgr2=0x7f4798135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f479813b1c0 0x7f4798135300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 -- 172.21.15.5:0/900933787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47980a2c80 msgr2=0x7f4798070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47980a2c80 0x7f4798070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient: _finish_hunting 0 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 monclient: found mon.noname-a 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 -- 172.21.15.5:0/900933787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790009570 con 0x7f47980a3770 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47967fc700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f478000e760 tx=0x7f478000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 client.?.objecter ms_handle_connect 0x7f47980a3770 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478000c070 con 0x7f47980a3770 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: dump: 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:57.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4780013610 con 0x7f47980a3770 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 5 monclient: authenticate success, global_id 74283 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _renew_subs 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47981211b0 con 0x7f47980a3770 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780013980 con 0x7f47980a3770 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 monclient: dump: 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.464 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f47957fa700 4 set_mon_vals no callback set 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 client.74283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _renew_subs 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47981359a0 con 0x7f47980a3770 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 1 librados: init done 2024-09-16T11:01:57.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4780013ae0 con 0x7f47980a3770 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 --2- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 0x7f4778039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(483..483 src has 344..483) v4 ==== 5298+0+0 (secure 0 0 0) 0x7f4780047b90 con 0x7f47980a3770 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 client.74283.objecter ms_dispatch 0x7f4798000fd0 osd_map(483..483 src has 344..483) v4 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 3 client.74283.objecter handle_osd_map got epochs [483,483] > 0 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 3 client.74283.objecter handle_osd_map decoding full epoch 483 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 20 client.74283.objecter dump_active .. 0 homeless 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47980a3060 con 0x7f47980a3770 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 1 --2- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 0x7f4778039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 10 monclient: get_auth_request con 0x7f4778036f60 auth_method 0 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f4796ffd700 1 --2- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 0x7f4778039420 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7f478800f290 tx=0x7f4788009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.446+0000 7f478f7fe700 10 client.74283.objecter ms_handle_connect 0x7f4778036f60 2024-09-16T11:01:57.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.454+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f478000f040 con 0x7f47980a3770 2024-09-16T11:01:57.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.454+0000 7f478f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:57.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.454+0000 7f478f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.690+0000 7f4797fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.690+0000 7f4797fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:57.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.690+0000 7f4797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:57.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.690+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f47980721d0 con 0x7f47980a3770 2024-09-16T11:01:57.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f478f7fe700 1 -- 172.21.15.5:0/900933787 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f47800106e0 con 0x7f47980a3770 2024-09-16T11:01:57.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f478f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:57.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f478f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:57.699 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 10 librados: watch_flush enter 2024-09-16T11:01:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 10 librados: watch_flush exit 2024-09-16T11:01:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 20 client.74283.objecter shutdown clearing up homeless session... 2024-09-16T11:01:57.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 10 client.74283.objecter successfully canceled tick 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 msgr2=0x7f4778039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 0x7f4778039420 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7f478800f290 tx=0x7f4788009040 comp rx=0 tx=0).stop 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 10 monclient: shutdown 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 msgr2=0x7f4798134dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f478000e760 tx=0x7f478000ea70 comp rx=0 tx=0).stop 2024-09-16T11:01:57.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 shutdown_connections 2024-09-16T11:01:57.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f479813b1c0 0x7f4798135300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4778036f60 0x7f4778039420 unknown :-1 s=CLOSED pgs=3848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f47980a3770 0x7f4798134dc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 --2- 172.21.15.5:0/900933787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47980a2c80 0x7f4798070520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 >> 172.21.15.5:0/900933787 conn(0x7f47980a4d80 msgr2=0x7f4798135c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 shutdown_connections 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 -- 172.21.15.5:0/900933787 wait complete. 2024-09-16T11:01:57.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.694+0000 7f4797fff700 1 librados: shutdown 2024-09-16T11:01:57.713 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:01:57.713 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T11:01:57.713 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:01:57.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: get_monmap_and_config 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: build_initial_monmap 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: monmap: 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:57.830931+0000 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:57.830931+0000 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.832 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:57.833 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:57.833 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 Processor -- start 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 -- start start 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: init 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a0c070750 addr v1:172.21.15.143:6789/0 2024-09-16T11:01:57.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a0c0a04b0 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c13e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c09fb60 con 0x7f2a0c0a04b0 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0a0090 con 0x7f2a0c070750 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c09f7f0 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a10932700 10 monclient(hunting): authenticate will time out at 1576.407562s 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0a59c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2a0c0a04b0 0x7f2a0c13ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47124/0 (socket says 172.21.15.5:47124) 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0a59c700 1 -- 172.21.15.5:0/1093196482 learned_addr learned my addr 172.21.15.5:0/1093196482 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0b59e700 1 --2- 172.21.15.5:0/1093196482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c13e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0b59e700 10 monclient(hunting): get_auth_request con 0x7f2a0c0a14c0 auth_method 0 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.826+0000 7f2a0b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_done global_id 84137 payload 274 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 1 -- 172.21.15.5:0/1093196482 >> v1:172.21.15.143:6789/0 conn(0x7f2a0c070750 legacy=0x7f2a0c070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 1 -- 172.21.15.5:0/1093196482 >> v1:172.21.15.5:6789/0 conn(0x7f2a0c0a04b0 legacy=0x7f2a0c13ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient: _finish_hunting 0 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 1 monclient: found mon.noname-b 2024-09-16T11:01:57.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 1 -- 172.21.15.5:0/1093196482 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c09ad40 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 1 -- 172.21.15.5:0/1093196482 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663386055 0 0) 0x7f2a0c09fb60 con 0x7f2a0c0a04b0 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a0b59e700 1 --2- 172.21.15.5:0/1093196482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c13e4b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009dc0 tx=0x7f2a0400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8632bee9425ebf8c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 1 -- 172.21.15.5:0/1093196482 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400e070 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: dump: 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.840 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: _finish_auth 0 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: _check_auth_tickets 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 5 monclient: authenticate success, global_id 84137 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:57.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 1 -- 172.21.15.5:0/1093196482 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400bbd0 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 1 -- 172.21.15.5:0/1093196482 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04012430 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 4 set_mon_vals no callback set 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a09d9b700 10 monclient: dump: 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.842 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 10 monclient: shutdown 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 -- 172.21.15.5:0/1093196482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 msgr2=0x7f2a0c13e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 --2- 172.21.15.5:0/1093196482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c13e4b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009dc0 tx=0x7f2a0400b330 comp rx=0 tx=0).stop 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 -- 172.21.15.5:0/1093196482 shutdown_connections 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 --2- 172.21.15.5:0/1093196482 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c13e4b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 -- 172.21.15.5:0/1093196482 >> 172.21.15.5:0/1093196482 conn(0x7f2a0c0a45d0 msgr2=0x7f2a0c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:57.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 -- 172.21.15.5:0/1093196482 shutdown_connections 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 1 -- 172.21.15.5:0/1093196482 wait complete. 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 10 monclient: build_initial_monmap 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.830+0000 7f2a10932700 10 monclient: monmap: 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:57.844 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 librados: starting msgr at 2024-09-16T11:01:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 librados: starting objecter 2024-09-16T11:01:57.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 Processor -- start 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- start start 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 librados: setting wanted keys 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 librados: calling monclient init 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: init 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c070750 0x7f2a0c1caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c070750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c0a04b0 0x7f2a0c1cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c0a04b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:57.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c0a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c09b0d0 con 0x7f2a0c0a04b0 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c11ee70 con 0x7f2a0c070750 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c11d750 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): authenticate will time out at 1576.413299s 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33550/0 (socket says 172.21.15.5:33550) 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 -- 172.21.15.5:0/2632944312 learned_addr learned my addr 172.21.15.5:0/2632944312 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:57.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): get_auth_request con 0x7f2a0c0a14c0 auth_method 0 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c0a04b0 0x7f2a0c1cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 10 monclient(hunting): get_auth_request con 0x7f2a0c0a04b0 auth_method 0 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0ad9d700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c070750 0x7f2a0c1caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0ad9d700 10 monclient(hunting): get_auth_request con 0x7f2a0c070750 auth_method 0 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient(hunting): handle_auth_done global_id 84140 payload 824 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c070750 msgr2=0x7f2a0c1caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a0c070750 0x7f2a0c1caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c0a04b0 msgr2=0x7f2a0c1cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c0a04b0 0x7f2a0c1cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient: _finish_hunting 0 2024-09-16T11:01:57.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 monclient: found mon.noname-c 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04009a70 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0a59c700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c0a04b0 0x7f2a0c1cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0b59e700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a04014040 tx=0x7f2a04004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 client.?.objecter ms_handle_connect 0x7f2a0c0a14c0 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0401b960 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: dump: 2024-09-16T11:01:57.854 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: _finish_auth 0 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: _check_auth_tickets 2024-09-16T11:01:57.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 5 monclient: authenticate success, global_id 84140 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a04020070 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0401bd40 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _renew_subs 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c13e4d0 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a0959a700 4 set_mon_vals no callback set 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f29f37fe700 10 monclient: dump: 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.856 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 client.84140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _renew_subs 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c0a1fe0 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 librados: init done 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 0x7f2a0c11bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.834+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c11f000 con 0x7f2a0c071240 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 0x7f2a0c11bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2a0401c400 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient: get_auth_request con 0x7f2a0c071240 auth_method 0 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 --2- 172.21.15.5:0/2632944312 >> v2:172.21.15.143:6812/1098205034 conn(0x7f29f8036f60 0x7f29f8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(483..483 src has 344..483) v4 ==== 5298+0+0 (secure 0 0 0) 0x7f2a04049810 con 0x7f2a0c0a14c0 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 client.84140.objecter ms_dispatch 0x7f2a0c000fd0 osd_map(483..483 src has 344..483) v4 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 3 client.84140.objecter handle_osd_map got epochs [483,483] > 0 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 3 client.84140.objecter handle_osd_map decoding full epoch 483 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 20 client.84140.objecter dump_active .. 0 homeless 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_done global_id 74292 payload 824 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0a59c700 1 --2- 172.21.15.5:0/2632944312 >> v2:172.21.15.143:6812/1098205034 conn(0x7f29f8036f60 0x7f29f8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a0c070b30 con 0x7f2a0c071240 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0a59c700 10 monclient: get_auth_request con 0x7f29f8036f60 auth_method 0 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 client.84140.objecter ms_handle_connect 0x7f2a0c071240 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 client.84140.objecter resend_mon_ops 2024-09-16T11:01:57.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0ad9d700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 0x7f2a0c11bc50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000e700 tx=0x7f2a0000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a000052e0 con 0x7f2a0c071240 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 monclient: dump: 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.860 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 client.84140.objecter ms_handle_connect 0x7f29f8036f60 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f2a0a59c700 1 --2- 172.21.15.5:0/2632944312 >> v2:172.21.15.143:6812/1098205034 conn(0x7f29f8036f60 0x7f29f8039420 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c11e480 tx=0x7f29fc007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2a00005440 con 0x7f2a0c071240 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 msgr2=0x7f2a0c11bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 0x7f2a0c11bc50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000e700 tx=0x7f2a0000eac0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 msgr2=0x7f2a0c11bc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:57.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.838+0000 7f29f37fe700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c071240 0x7f2a0c11bc50 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000e700 tx=0x7f2a0000eac0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a10932700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a10932700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 0x7f2a0c02bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a10932700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c067380 con 0x7f2a0c02b6a0 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 0x7f2a0c02bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient: get_auth_request con 0x7f2a0c02b6a0 auth_method 0 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 10 monclient(hunting): handle_auth_done global_id 74295 payload 824 2024-09-16T11:01:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 1 -- 172.21.15.5:0/2632944312 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a0c13a530 con 0x7f2a0c02b6a0 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f2a0ad9d700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 0x7f2a0c02bab0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a000013f0 tx=0x7f2a00001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 client.84140.objecter ms_handle_connect 0x7f2a0c02b6a0 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 client.84140.objecter resend_mon_ops 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00022070 con 0x7f2a0c02b6a0 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 monclient: dump: 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f2a00006fb0 con 0x7f2a0c02b6a0 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 msgr2=0x7f2a0c02bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 0x7f2a0c02bab0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a000013f0 tx=0x7f2a00001120 comp rx=0 tx=0).stop 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 msgr2=0x7f2a0c02bab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:57.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.850+0000 7f29f37fe700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a0c02b6a0 0x7f2a0c02bab0 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2a000013f0 tx=0x7f2a00001120 comp rx=0 tx=0).stop 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:57.867 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:57.868 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:57.869 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:57.870 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:57.871 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:57.872 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.873 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.874 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:57.875 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:57.877 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:57.878 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:57.879 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 10 librados: watch_flush enter 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 10 librados: watch_flush exit 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 20 client.84140.objecter shutdown clearing up homeless session... 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 10 client.84140.objecter successfully canceled tick 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 >> v2:172.21.15.143:6812/1098205034 conn(0x7f29f8036f60 msgr2=0x7f29f8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 --2- 172.21.15.5:0/2632944312 >> v2:172.21.15.143:6812/1098205034 conn(0x7f29f8036f60 0x7f29f8039420 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c11e480 tx=0x7f29fc007d00 comp rx=0 tx=0).stop 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 10 monclient: shutdown 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 msgr2=0x7f2a0c11af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a0b59e700 1 -- 172.21.15.5:0/2632944312 reap_dead start 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a04014040 tx=0x7f2a04004af0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 shutdown_connections 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 --2- 172.21.15.5:0/2632944312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2a0c0a14c0 0x7f2a0c11af40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 >> 172.21.15.5:0/2632944312 conn(0x7f2a0c0a45d0 msgr2=0x7f2a0c0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 shutdown_connections 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 -- 172.21.15.5:0/2632944312 wait complete. 2024-09-16T11:01:57.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.854+0000 7f2a10932700 1 librados: shutdown 2024-09-16T11:01:57.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:57.883 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:01:57.883 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.990+0000 7faff11df700 10 monclient: get_monmap_and_config 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.990+0000 7faff11df700 10 monclient: build_initial_monmap 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.990+0000 7faff11df700 10 monclient: monmap: 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:57.996834+0000 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:57.996834+0000 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:57.998 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:57.999 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:57.999 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:57.999 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:57.999 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 Processor -- start 2024-09-16T11:01:58.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 -- start start 2024-09-16T11:01:58.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient: init 2024-09-16T11:01:58.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 0x7fafec09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): picked mon.noname-b con 0x7fafec09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): picked mon.noname-a con 0x7fafec09d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafec09dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafec070e40 con 0x7fafec09dfd0 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafec0a01d0 con 0x7fafec09d6a0 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec09f740 con 0x7fafec09e8e0 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient(hunting): authenticate will time out at 1576.573255s 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 0x7fafec09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39680/0 (socket says 172.21.15.5:39680) 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 1 -- 172.21.15.5:0/1016698560 learned_addr learned my addr 172.21.15.5:0/1016698560 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): get_auth_request con 0x7fafec09d6a0 auth_method 0 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): get_auth_request con 0x7fafec09e8e0 auth_method 0 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient(hunting): handle_auth_done global_id 84143 payload 274 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 -- 172.21.15.5:0/1016698560 >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 msgr2=0x7fafec09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 --2- 172.21.15.5:0/1016698560 >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 -- 172.21.15.5:0/1016698560 >> v1:172.21.15.5:6789/0 conn(0x7fafec09dfd0 legacy=0x7fafec1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 monclient: found mon.noname-b 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 -- 172.21.15.5:0/1016698560 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec09fef0 con 0x7fafec09e8e0 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeb7fe700 1 --2- 172.21.15.5:0/1016698560 >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafebfff700 1 --2- 172.21.15.5:0/1016698560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 0x7fafec09ecc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009850 tx=0x7fafd4009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0ecfe610ab0816c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 1 -- 172.21.15.5:0/1016698560 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4011830 con 0x7fafec09e8e0 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: dump: 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.006 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 1 -- 172.21.15.5:0/1016698560 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd4011990 con 0x7fafec09e8e0 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 5 monclient: authenticate success, global_id 84143 2024-09-16T11:01:58.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 1 -- 172.21.15.5:0/1016698560 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4011d40 con 0x7fafec09e8e0 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7faff11df700 4 set_mon_vals no callback set 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.994+0000 7fafeaffd700 10 monclient: dump: 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.008 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: shutdown 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- 172.21.15.5:0/1016698560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 msgr2=0x7fafec09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- 172.21.15.5:0/1016698560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 0x7fafec09ecc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009850 tx=0x7fafd4009b60 comp rx=0 tx=0).stop 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- 172.21.15.5:0/1016698560 shutdown_connections 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- 172.21.15.5:0/1016698560 >> v2:172.21.15.5:3300/0 conn(0x7fafec09d6a0 0x7fafec09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- 172.21.15.5:0/1016698560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09e8e0 0x7fafec09ecc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- 172.21.15.5:0/1016698560 >> 172.21.15.5:0/1016698560 conn(0x7fafec0a45d0 msgr2=0x7fafec0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- 172.21.15.5:0/1016698560 shutdown_connections 2024-09-16T11:01:58.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- 172.21.15.5:0/1016698560 wait complete. 2024-09-16T11:01:58.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: monmap: 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 librados: starting msgr at 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 librados: starting objecter 2024-09-16T11:01:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 Processor -- start 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 -- start start 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 librados: setting wanted keys 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 librados: calling monclient init 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: init 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafec09d6a0 0x7fafec132980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient(hunting): picked mon.noname-b con 0x7fafec09d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09dfd0 0x7fafec132ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient(hunting): picked mon.noname-c con 0x7fafec09dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafec09e8e0 0x7fafec11c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:57.998+0000 7faff11df700 10 monclient(hunting): picked mon.noname-a con 0x7fafec09e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:58.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec11fe60 con 0x7fafec09e8e0 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec1c90c0 con 0x7fafec09d6a0 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec120340 con 0x7fafec09dfd0 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): authenticate will time out at 1576.580051s 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09dfd0 0x7fafec132ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09dfd0 0x7fafec132ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33570/0 (socket says 172.21.15.5:33570) 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 1 -- 172.21.15.5:0/3205797270 learned_addr learned my addr 172.21.15.5:0/3205797270 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): get_auth_request con 0x7fafec09dfd0 auth_method 0 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafec09e8e0 0x7fafec11c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafec09d6a0 0x7fafec132980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): get_auth_request con 0x7fafec09e8e0 auth_method 0 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): get_auth_request con 0x7fafec09d6a0 auth_method 0 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient(hunting): handle_auth_done global_id 74301 payload 824 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09dfd0 msgr2=0x7fafec132ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec09dfd0 0x7fafec132ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafec09d6a0 msgr2=0x7fafec132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafec09d6a0 0x7fafec132980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 monclient: found mon.noname-a 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc009b30 con 0x7fafec09e8e0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 client.?.objecter ms_handle_connect 0x7fafec09e8e0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff09de700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafec09e8e0 0x7fafec11c410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fafe4001eb0 tx=0x7fafe400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40052d0 con 0x7fafec09e8e0 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: dump: 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 5 monclient: authenticate success, global_id 74301 2024-09-16T11:01:58.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4005430 con 0x7fafec09e8e0 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40055a0 con 0x7fafec09e8e0 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe9ffb700 4 set_mon_vals no callback set 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 monclient: dump: 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient: _renew_subs 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd40094b0 con 0x7fafec09e8e0 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 client.74301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient: _renew_subs 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec09b0d0 con 0x7fafec09e8e0 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 librados: init done 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fafe40038e0 con 0x7fafec09e8e0 2024-09-16T11:01:58.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 0x7fafec11c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec067380 con 0x7fafec1c8410 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 --2- 172.21.15.5:0/3205797270 >> v2:172.21.15.143:6812/1098205034 conn(0x7fafcc036f60 0x7fafcc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 0x7fafec11c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(483..483 src has 344..483) v4 ==== 5298+0+0 (secure 0 0 0) 0x7fafe4047060 con 0x7fafec09e8e0 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 10 client.74301.objecter ms_dispatch 0x7fafec000fd0 osd_map(483..483 src has 344..483) v4 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 3 client.74301.objecter handle_osd_map got epochs [483,483] > 0 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 3 client.74301.objecter handle_osd_map decoding full epoch 483 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient: get_auth_request con 0x7fafec1c8410 auth_method 0 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafe8ff9700 20 client.74301.objecter dump_active .. 0 homeless 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 1 --2- 172.21.15.5:0/3205797270 >> v2:172.21.15.143:6812/1098205034 conn(0x7fafcc036f60 0x7fafcc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafeb7fe700 10 monclient: get_auth_request con 0x7fafcc036f60 auth_method 0 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 10 monclient(hunting): handle_auth_done global_id 84149 payload 824 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.002+0000 7fafebfff700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafec133dd0 con 0x7fafec1c8410 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafebfff700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 0x7fafec11c950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009850 tx=0x7fafd4004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 client.74301.objecter ms_handle_connect 0x7fafec1c8410 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 client.74301.objecter resend_mon_ops 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafeb7fe700 1 --2- 172.21.15.5:0/3205797270 >> v2:172.21.15.143:6812/1098205034 conn(0x7fafcc036f60 0x7fafcc039420 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7fafec11dbe0 tx=0x7fafdc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 client.74301.objecter ms_handle_connect 0x7fafcc036f60 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4020070 con 0x7fafec1c8410 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 monclient: dump: 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fafd4004330 con 0x7fafec1c8410 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 msgr2=0x7fafec11c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 0x7fafec11c950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009850 tx=0x7fafd4004770 comp rx=0 tx=0).stop 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 msgr2=0x7fafec11c950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.006+0000 7fafe8ff9700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec1c8410 0x7fafec11c950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.014+0000 7faff11df700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.014+0000 7faff11df700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 0x7fafec06d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.014+0000 7faff11df700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.014+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafec11d2a0 con 0x7fafec11ce90 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 0x7fafec06d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient: get_auth_request con 0x7fafec11ce90 auth_method 0 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 10 monclient(hunting): handle_auth_done global_id 84152 payload 824 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 1 -- 172.21.15.5:0/3205797270 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fafec1341b0 con 0x7fafec11ce90 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 client.74301.objecter ms_handle_connect 0x7fafec11ce90 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 client.74301.objecter resend_mon_ops 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafebfff700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 0x7fafec06d4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd402eae0 con 0x7fafec11ce90 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 monclient: dump: 2024-09-16T11:01:58.031 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7fafd4004580 con 0x7fafec11ce90 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 msgr2=0x7fafec06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 0x7fafec06d4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4009240 comp rx=0 tx=0).stop 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 msgr2=0x7fafec06d4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7fafe8ff9700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fafec11ce90 0x7fafec06d4c0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4009240 comp rx=0 tx=0).stop 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:58.033 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.034 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.035 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:58.036 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.037 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.038 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.039 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.040 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.041 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.042 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.043 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:58.044 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:58.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 10 librados: watch_flush enter 2024-09-16T11:01:58.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 10 librados: watch_flush exit 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 20 client.74301.objecter shutdown clearing up homeless session... 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 10 client.74301.objecter successfully canceled tick 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 >> v2:172.21.15.143:6812/1098205034 conn(0x7fafcc036f60 msgr2=0x7fafcc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 --2- 172.21.15.5:0/3205797270 >> v2:172.21.15.143:6812/1098205034 conn(0x7fafcc036f60 0x7fafcc039420 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7fafec11dbe0 tx=0x7fafdc00b040 comp rx=0 tx=0).stop 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 10 monclient: shutdown 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafec09e8e0 msgr2=0x7fafec11c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 --2- 172.21.15.5:0/3205797270 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fafec09e8e0 0x7fafec11c410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fafe4001eb0 tx=0x7fafe400c6e0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff09de700 1 -- 172.21.15.5:0/3205797270 reap_dead start 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff09de700 1 -- 172.21.15.5:0/3205797270 reap_dead start 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 shutdown_connections 2024-09-16T11:01:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 >> 172.21.15.5:0/3205797270 conn(0x7fafec0a45d0 msgr2=0x7fafec0a3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 shutdown_connections 2024-09-16T11:01:58.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 -- 172.21.15.5:0/3205797270 wait complete. 2024-09-16T11:01:58.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.018+0000 7faff11df700 1 librados: shutdown 2024-09-16T11:01:58.048 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:58.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:01:58.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:01:58.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 10 monclient: get_monmap_and_config 2024-09-16T11:01:58.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 10 monclient: monmap: 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:58.162681+0000 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:58.162681+0000 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:58.164 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 1 Processor -- start 2024-09-16T11:01:58.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 1 -- start start 2024-09-16T11:01:58.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.158+0000 7f441f01a700 10 monclient: init 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44180a36b0 addr v1:172.21.15.5:6789/0 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f441806cc10 0x7f441806f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-a con 0x7f441806cc10 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 0x7f44181391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-b con 0x7f44180a41a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441806d090 con 0x7f44180a36b0 2024-09-16T11:01:58.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441806e130 con 0x7f441806cc10 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441806e9d0 con 0x7f44180a41a0 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient(hunting): authenticate will time out at 1576.740336s 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f441806cc10 0x7f441806f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 0x7f44181391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e018700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f44180a36b0 0x7f44180a3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47132/0 (socket says 172.21.15.5:47132) 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e018700 1 -- 172.21.15.5:0/2283945401 learned_addr learned my addr 172.21.15.5:0/2283945401 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): get_auth_request con 0x7f441806cc10 auth_method 0 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): get_auth_request con 0x7f44180a41a0 auth_method 0 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient(hunting): handle_auth_done global_id 84155 payload 274 2024-09-16T11:01:58.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 -- 172.21.15.5:0/2283945401 >> v2:172.21.15.5:3300/0 conn(0x7f441806cc10 msgr2=0x7f441806f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 --2- 172.21.15.5:0/2283945401 >> v2:172.21.15.5:3300/0 conn(0x7f441806cc10 0x7f441806f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 -- 172.21.15.5:0/2283945401 >> v1:172.21.15.5:6789/0 conn(0x7f44180a36b0 legacy=0x7f44180a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 monclient: found mon.noname-b 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 -- 172.21.15.5:0/2283945401 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44180a2030 con 0x7f44180a41a0 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441e819700 1 --2- 172.21.15.5:0/2283945401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 0x7f44181391b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4410009850 tx=0x7f4410009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=502a8755ef8d79b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 1 -- 172.21.15.5:0/2283945401 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410011830 con 0x7f44180a41a0 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: dump: 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 5 monclient: authenticate success, global_id 84155 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 1 -- 172.21.15.5:0/2283945401 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410011990 con 0x7f44180a41a0 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 4 set_mon_vals no callback set 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 1 -- 172.21.15.5:0/2283945401 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410011d40 con 0x7f44180a41a0 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441d016700 10 monclient: dump: 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.174 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 10 monclient: shutdown 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- 172.21.15.5:0/2283945401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 msgr2=0x7f44181391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 --2- 172.21.15.5:0/2283945401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 0x7f44181391b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4410009850 tx=0x7f4410009b60 comp rx=0 tx=0).stop 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- 172.21.15.5:0/2283945401 shutdown_connections 2024-09-16T11:01:58.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 --2- 172.21.15.5:0/2283945401 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a41a0 0x7f44181391b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 --2- 172.21.15.5:0/2283945401 >> v2:172.21.15.5:3300/0 conn(0x7f441806cc10 0x7f441806f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- 172.21.15.5:0/2283945401 >> 172.21.15.5:0/2283945401 conn(0x7f44180a12a0 msgr2=0x7f44180a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- 172.21.15.5:0/2283945401 shutdown_connections 2024-09-16T11:01:58.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.162+0000 7f441f01a700 1 -- 172.21.15.5:0/2283945401 wait complete. 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: monmap: 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:58.181 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 librados: starting msgr at 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 librados: starting objecter 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 Processor -- start 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 -- start start 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 librados: setting wanted keys 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 librados: calling monclient init 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: init 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806cc10 0x7f4418131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-a con 0x7f441806cc10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 0x7f4418131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-c con 0x7f44180a36b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44180a41a0 0x7f4418136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): picked mon.noname-b con 0x7f44180a41a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4418071860 con 0x7f441806cc10 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441813a970 con 0x7f44180a41a0 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441813a720 con 0x7f44180a36b0 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient(hunting): authenticate will time out at 1576.746556s 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 0x7f4418131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806cc10 0x7f4418131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 0x7f4418131fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33594/0 (socket says 172.21.15.5:33594) 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 -- 172.21.15.5:0/712220254 learned_addr learned my addr 172.21.15.5:0/712220254 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): get_auth_request con 0x7f44180a36b0 auth_method 0 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): get_auth_request con 0x7f441806cc10 auth_method 0 2024-09-16T11:01:58.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient(hunting): handle_auth_done global_id 84158 payload 824 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44180a41a0 msgr2=0x7f4418136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44180a41a0 0x7f4418136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806cc10 msgr2=0x7f4418131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806cc10 0x7f4418131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 monclient: found mon.noname-c 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44100094b0 con 0x7f44180a36b0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441e018700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806cc10 0x7f4418131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 client.?.objecter ms_handle_connect 0x7f44180a36b0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441d817700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 0x7f4418131fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f440c00e9d0 tx=0x7f440c00ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c009720 con 0x7f44180a36b0 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: dump: 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 5 monclient: authenticate success, global_id 84158 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c00c850 con 0x7f44180a36b0 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: _renew_subs 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c00cc00 con 0x7f44180a36b0 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441809e5c0 con 0x7f44180a36b0 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.166+0000 7f441c815700 4 set_mon_vals no callback set 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: dump: 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.188 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 10 client.84158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 10 monclient: _renew_subs 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44181391d0 con 0x7f44180a36b0 2024-09-16T11:01:58.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 1 librados: init done 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f440c00cd60 con 0x7f44180a36b0 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 0x7f44181372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441813ab00 con 0x7f4418072460 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 1 --2- 172.21.15.5:0/712220254 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4404036f60 0x7f4404039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(484..484 src has 344..484) v4 ==== 5556+0+0 (secure 0 0 0) 0x7f440c047ef0 con 0x7f44180a36b0 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 client.84158.objecter ms_dispatch 0x7f4418000fd0 osd_map(484..484 src has 344..484) v4 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 3 client.84158.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 3 client.84158.objecter handle_osd_map decoding full epoch 484 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 20 client.84158.objecter dump_active .. 0 homeless 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 0x7f44181372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient: get_auth_request con 0x7f4418072460 auth_method 0 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e819700 1 --2- 172.21.15.5:0/712220254 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4404036f60 0x7f4404039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e819700 10 monclient: get_auth_request con 0x7f4404036f60 auth_method 0 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 10 monclient(hunting): handle_auth_done global_id 64408 payload 824 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441809d170 con 0x7f4418072460 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 client.84158.objecter ms_handle_connect 0x7f4418072460 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 client.84158.objecter resend_mon_ops 2024-09-16T11:01:58.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e018700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 0x7f44181372a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f44080012e0 tx=0x7f440800d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 client.84158.objecter ms_handle_connect 0x7f4404036f60 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f441e819700 1 --2- 172.21.15.5:0/712220254 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4404036f60 0x7f4404039420 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f4418132970 tx=0x7f4410010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440800dce0 con 0x7f4418072460 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.170+0000 7f4416ffd700 10 monclient: dump: 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.192 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f440800de40 con 0x7f4418072460 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 msgr2=0x7f44181372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 0x7f44181372a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f44080012e0 tx=0x7f440800d430 comp rx=0 tx=0).stop 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 msgr2=0x7f44181372a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.174+0000 7f4416ffd700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4418072460 0x7f44181372a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441f01a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441f01a700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 0x7f4418137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441f01a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4418074c30 con 0x7f44181377e0 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 0x7f4418137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient: get_auth_request con 0x7f44181377e0 auth_method 0 2024-09-16T11:01:58.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 10 monclient(hunting): handle_auth_done global_id 64411 payload 824 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 1 -- 172.21.15.5:0/712220254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f44180a0a40 con 0x7f44181377e0 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 client.84158.objecter ms_handle_connect 0x7f44181377e0 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 client.84158.objecter resend_mon_ops 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f441e018700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 0x7f4418137bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44080012e0 tx=0x7f4408005c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408023070 con 0x7f44181377e0 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 monclient: dump: 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f440801e320 con 0x7f44181377e0 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:01:58.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 msgr2=0x7f4418137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 0x7f4418137bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44080012e0 tx=0x7f4408005c20 comp rx=0 tx=0).stop 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 msgr2=0x7f4418137bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.186+0000 7f4416ffd700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44181377e0 0x7f4418137bf0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44080012e0 tx=0x7f4408005c20 comp rx=0 tx=0).stop 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 48, 2024-09-16T11:01:58.198 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 4, 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.199 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.200 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:01:58.201 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:01:58.202 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.203 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:01:58.204 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.205 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.206 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:01:58.207 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:01:58.208 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.209 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:01:58.210 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:01:58.211 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 10 librados: watch_flush enter 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 10 librados: watch_flush exit 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 20 client.84158.objecter shutdown clearing up homeless session... 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 10 client.84158.objecter successfully canceled tick 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4404036f60 msgr2=0x7f4404039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 --2- 172.21.15.5:0/712220254 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4404036f60 0x7f4404039420 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f4418132970 tx=0x7f4410010000 comp rx=0 tx=0).stop 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 10 monclient: shutdown 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 msgr2=0x7f4418131fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 --2- 172.21.15.5:0/712220254 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f44180a36b0 0x7f4418131fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f440c00e9d0 tx=0x7f440c00ece0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441e819700 1 -- 172.21.15.5:0/712220254 reap_dead start 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441e819700 1 -- 172.21.15.5:0/712220254 reap_dead start 2024-09-16T11:01:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 shutdown_connections 2024-09-16T11:01:58.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 >> 172.21.15.5:0/712220254 conn(0x7f44180a12a0 msgr2=0x7f441809dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 shutdown_connections 2024-09-16T11:01:58.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 -- 172.21.15.5:0/712220254 wait complete. 2024-09-16T11:01:58.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.190+0000 7f441f01a700 1 librados: shutdown 2024-09-16T11:01:58.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:01:58.214 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T11:01:58.214 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T11:01:58.214 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T11:01:58.215 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-16T11:01:58.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.318+0000 7fe38896d700 10 monclient: get_monmap_and_config 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.318+0000 7fe38896d700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.318+0000 7fe38896d700 10 monclient: monmap: 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:58.324053+0000 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:58.324053+0000 2024-09-16T11:01:58.325 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:58.326 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 Processor -- start 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- start start 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient: init 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe3840a2dd0 0x7fe3840a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-c con 0x7fe3840a2dd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 0x7fe3840a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-a con 0x7fe3840a36f0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 0x7fe384136040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-b con 0x7fe384135b90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3840a4d60 con 0x7fe3840a36f0 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3840a4a10 con 0x7fe3840a2dd0 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38409f150 con 0x7fe384135b90 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient(hunting): authenticate will time out at 1576.900970s 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 0x7fe384136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 0x7fe3840a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 0x7fe3840a3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39706/0 (socket says 172.21.15.5:39706) 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 1 -- 172.21.15.5:0/739986193 learned_addr learned my addr 172.21.15.5:0/739986193 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): get_auth_request con 0x7fe384135b90 auth_method 0 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): get_auth_request con 0x7fe3840a36f0 auth_method 0 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 1 --2- 172.21.15.5:0/739986193 >> v2:172.21.15.143:3300/0 conn(0x7fe3840a2dd0 0x7fe3840a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): get_auth_request con 0x7fe3840a2dd0 auth_method 0 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient(hunting): handle_auth_done global_id 84161 payload 274 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 -- 172.21.15.5:0/739986193 >> v2:172.21.15.143:3300/0 conn(0x7fe3840a2dd0 msgr2=0x7fe3840a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 --2- 172.21.15.5:0/739986193 >> v2:172.21.15.143:3300/0 conn(0x7fe3840a2dd0 0x7fe3840a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 -- 172.21.15.5:0/739986193 >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 msgr2=0x7fe3840a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:58.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 --2- 172.21.15.5:0/739986193 >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 0x7fe3840a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 monclient: found mon.noname-b 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 -- 172.21.15.5:0/739986193 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38409f390 con 0x7fe384135b90 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe383fff700 1 --2- 172.21.15.5:0/739986193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 0x7fe384136040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe3840a5f90 tx=0x7fe37c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7d74537acb0985 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 1 -- 172.21.15.5:0/739986193 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c00e070 con 0x7fe384135b90 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: dump: 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.340 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 5 monclient: authenticate success, global_id 84161 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:58.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 1 -- 172.21.15.5:0/739986193 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c00bb20 con 0x7fe384135b90 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 4 set_mon_vals no callback set 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 1 -- 172.21.15.5:0/739986193 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c012430 con 0x7fe384135b90 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe3827fc700 10 monclient: dump: 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.342 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 10 monclient: shutdown 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- 172.21.15.5:0/739986193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 msgr2=0x7fe384136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- 172.21.15.5:0/739986193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 0x7fe384136040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe3840a5f90 tx=0x7fe37c00b330 comp rx=0 tx=0).stop 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- 172.21.15.5:0/739986193 shutdown_connections 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- 172.21.15.5:0/739986193 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe384135b90 0x7fe384136040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- 172.21.15.5:0/739986193 >> v2:172.21.15.5:3300/0 conn(0x7fe3840a36f0 0x7fe3840a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 --2- 172.21.15.5:0/739986193 >> v2:172.21.15.143:3300/0 conn(0x7fe3840a2dd0 0x7fe3840a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- 172.21.15.5:0/739986193 >> 172.21.15.5:0/739986193 conn(0x7fe3840a4ed0 msgr2=0x7fe3840a52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- 172.21.15.5:0/739986193 shutdown_connections 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.322+0000 7fe38896d700 1 -- 172.21.15.5:0/739986193 wait complete. 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: monmap: 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:58.344 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 librados: starting msgr at 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 librados: starting objecter 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 Processor -- start 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- start start 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 librados: setting wanted keys 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 librados: calling monclient init 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: init 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3840a2dd0 0x7fe384070690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-a con 0x7fe3840a2dd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-c con 0x7fe3840a36f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe384135b90 0x7fe384134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): picked mon.noname-b con 0x7fe384135b90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3841213e0 con 0x7fe3840a2dd0 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3840712e0 con 0x7fe384135b90 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe384120040 con 0x7fe3840a36f0 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): authenticate will time out at 1576.905046s 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33616/0 (socket says 172.21.15.5:33616) 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 -- 172.21.15.5:0/2728228504 learned_addr learned my addr 172.21.15.5:0/2728228504 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): get_auth_request con 0x7fe3840a36f0 auth_method 0 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3840a2dd0 0x7fe384070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): get_auth_request con 0x7fe3840a2dd0 auth_method 0 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe384135b90 0x7fe384134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): get_auth_request con 0x7fe384135b90 auth_method 0 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe383fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient(hunting): handle_auth_done global_id 84164 payload 824 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe384135b90 msgr2=0x7fe384134520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe384135b90 0x7fe384134520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3840a2dd0 msgr2=0x7fe384070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3840a2dd0 0x7fe384070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 monclient: found mon.noname-c 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c009a70 con 0x7fe3840a36f0 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 client.?.objecter ms_handle_connect 0x7fe3840a36f0 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe382ffd700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe37400e960 tx=0x7fe37400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3740096c0 con 0x7fe3840a36f0 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: dump: 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.350 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe374013680 con 0x7fe3840a36f0 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 5 monclient: authenticate success, global_id 84164 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _renew_subs 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3841ca850 con 0x7fe3840a36f0 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374013a30 con 0x7fe3840a36f0 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 10 monclient: dump: 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 client.84164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:58.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _renew_subs 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3841219a0 con 0x7fe3840a36f0 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 librados: init done 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe381ffb700 4 set_mon_vals no callback set 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 0x7fe38411ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe384067310 con 0x7fe38411ea20 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.326+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe374013b90 con 0x7fe3840a36f0 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 --2- 172.21.15.5:0/2728228504 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe370036f60 0x7fe370039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(484..484 src has 344..484) v4 ==== 5556+0+0 (secure 0 0 0) 0x7fe374047d80 con 0x7fe3840a36f0 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 client.84164.objecter ms_dispatch 0x7fe384000fd0 osd_map(484..484 src has 344..484) v4 2024-09-16T11:01:58.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 3 client.84164.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 3 client.84164.objecter handle_osd_map decoding full epoch 484 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 0x7fe38411ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 20 client.84164.objecter dump_active .. 0 homeless 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient: get_auth_request con 0x7fe38411ea20 auth_method 0 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe383fff700 1 --2- 172.21.15.5:0/2728228504 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe370036f60 0x7fe370039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe383fff700 10 monclient: get_auth_request con 0x7fe370036f60 auth_method 0 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 10 monclient(hunting): handle_auth_done global_id 64420 payload 824 2024-09-16T11:01:58.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3840a41a0 con 0x7fe38411ea20 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe3837fe700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 0x7fe38411ee00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378011ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 client.84164.objecter ms_handle_connect 0x7fe38411ea20 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 client.84164.objecter resend_mon_ops 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 client.84164.objecter ms_handle_connect 0x7fe370036f60 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe383fff700 1 --2- 172.21.15.5:0/2728228504 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe370036f60 0x7fe370039420 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7fe37c014040 tx=0x7fe37c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378004240 con 0x7fe38411ea20 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 monclient: dump: 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.355 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe3780043a0 con 0x7fe38411ea20 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 msgr2=0x7fe38411ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 0x7fe38411ee00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378011ce0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 msgr2=0x7fe38411ee00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.330+0000 7fe36bfff700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411ea20 0x7fe38411ee00 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378011ce0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe38896d700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe38896d700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 0x7fe38406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe38896d700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38409eed0 con 0x7fe38411f340 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 0x7fe38406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient: get_auth_request con 0x7fe38411f340 auth_method 0 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 10 monclient(hunting): handle_auth_done global_id 64423 payload 824 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.338+0000 7fe3837fe700 1 -- 172.21.15.5:0/2728228504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fe3840a5c60 con 0x7fe38411f340 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 client.84164.objecter ms_handle_connect 0x7fe38411f340 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 client.84164.objecter resend_mon_ops 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe3837fe700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 0x7fe38406c6e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37801e450 con 0x7fe38411f340 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 monclient: dump: 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (secure 0 0 0) 0x7fe37801e5b0 con 0x7fe38411f340 2024-09-16T11:01:58.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 msgr2=0x7fe38406c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 0x7fe38406c6e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378004770 comp rx=0 tx=0).stop 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 msgr2=0x7fe38406c6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe36bfff700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe38411f340 0x7fe38406c6e0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe378011920 tx=0x7fe378004770 comp rx=0 tx=0).stop 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stdout: "ret": 0, 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T11:01:58.361 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:01:58.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 10 librados: watch_flush enter 2024-09-16T11:01:58.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 10 librados: watch_flush exit 2024-09-16T11:01:58.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 20 client.84164.objecter shutdown clearing up homeless session... 2024-09-16T11:01:58.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 10 client.84164.objecter successfully canceled tick 2024-09-16T11:01:58.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe370036f60 msgr2=0x7fe370039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 --2- 172.21.15.5:0/2728228504 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe370036f60 0x7fe370039420 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7fe37c014040 tx=0x7fe37c011000 comp rx=0 tx=0).stop 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 10 monclient: shutdown 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 msgr2=0x7fe384133fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe37400e960 tx=0x7fe37400ec70 comp rx=0 tx=0).stop 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe383fff700 1 -- 172.21.15.5:0/2728228504 reap_dead start 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 shutdown_connections 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 --2- 172.21.15.5:0/2728228504 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe3840a36f0 0x7fe384133fe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 >> 172.21.15.5:0/2728228504 conn(0x7fe3840a4ed0 msgr2=0x7fe384134ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 shutdown_connections 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 -- 172.21.15.5:0/2728228504 wait complete. 2024-09-16T11:01:58.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.342+0000 7fe38896d700 1 librados: shutdown 2024-09-16T11:01:58.365 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T11:01:58.365 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T11:01:58.365 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:01:58.401 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T11:01:58.401 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T11:01:58.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:01:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: get_monmap_and_config 2024-09-16T11:01:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: monmap: 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:01:58.517454+0000 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:01:58.517454+0000 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:01:58.519 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 Processor -- start 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 -- start start 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: init 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 0x7ff3ec09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec09e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 0x7ff3ec09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec09d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec09dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ec0a01d0 con 0x7ff3ec09d6a0 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ec070e40 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec09f740 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient(hunting): authenticate will time out at 1577.093804s 2024-09-16T11:01:58.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f0d29700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 msgr2=0x7ff3ec09ecc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f0d29700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 0x7ff3ec09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 0x7ff3ec09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33624/0 (socket says 172.21.15.5:33624) 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 -- 172.21.15.5:0/2760976404 learned_addr learned my addr 172.21.15.5:0/2760976404 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): get_auth_request con 0x7ff3ec09dfd0 auth_method 0 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): get_auth_request con 0x7ff3ec09d6a0 auth_method 0 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient(hunting): handle_auth_done global_id 84167 payload 274 2024-09-16T11:01:58.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 -- 172.21.15.5:0/2760976404 >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 msgr2=0x7ff3ec09ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 --2- 172.21.15.5:0/2760976404 >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 0x7ff3ec09ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 -- 172.21.15.5:0/2760976404 >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 msgr2=0x7ff3ec09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 --2- 172.21.15.5:0/2760976404 >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 0x7ff3ec09f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 monclient: found mon.noname-b 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 -- 172.21.15.5:0/2760976404 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec09fef0 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e3fff700 1 --2- 172.21.15.5:0/2760976404 >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 0x7ff3ec09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f152a700 1 --2- 172.21.15.5:0/2760976404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cbf56e838ca8509 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 1 -- 172.21.15.5:0/2760976404 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400e070 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: dump: 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.529 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 1 -- 172.21.15.5:0/2760976404 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e400b8d0 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 5 monclient: authenticate success, global_id 84167 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 10 monclient: get_monmap_and_config success 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 1 -- 172.21.15.5:0/2760976404 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e400bc80 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3f1d2b700 4 set_mon_vals no callback set 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.514+0000 7ff3e37fe700 10 monclient: dump: 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: shutdown 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/2760976404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 msgr2=0x7ff3ec1336c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/2760976404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009e50 tx=0x7ff3e400b330 comp rx=0 tx=0).stop 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/2760976404 shutdown_connections 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/2760976404 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09dfd0 0x7ff3ec1336c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/2760976404 >> v2:172.21.15.5:3300/0 conn(0x7ff3ec09d6a0 0x7ff3ec09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/2760976404 >> v2:172.21.15.143:3300/0 conn(0x7ff3ec09e8e0 0x7ff3ec09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/2760976404 >> 172.21.15.5:0/2760976404 conn(0x7ff3ec0a45d0 msgr2=0x7ff3ec0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/2760976404 shutdown_connections 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/2760976404 wait complete. 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: build_initial_monmap 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: monmap: 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:01:58.532 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 librados: starting msgr at 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 librados: starting objecter 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 Processor -- start 2024-09-16T11:01:58.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- start start 2024-09-16T11:01:58.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 librados: setting wanted keys 2024-09-16T11:01:58.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 librados: calling monclient init 2024-09-16T11:01:58.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: init 2024-09-16T11:01:58.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: _reopen_session rank -1 2024-09-16T11:01:58.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:01:58.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 0x7ff3ec1cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec09d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ec09dfd0 0x7ff3ec1cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec09dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec09d360 con 0x7ff3ec09dfd0 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1234a0 con 0x7ff3ec09d6a0 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1239d0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): _renew_subs 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f1d2b700 10 monclient(hunting): authenticate will time out at 1577.099337s 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f0d29700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 msgr2=0x7ff3ec1cabf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f0d29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 0x7ff3ec1cabf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ec09dfd0 0x7ff3ec1cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33638/0 (socket says 172.21.15.5:33638) 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 1 -- 172.21.15.5:0/1586624742 learned_addr learned my addr 172.21.15.5:0/1586624742 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): get_auth_request con 0x7ff3ec09e8e0 auth_method 0 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): get_auth_request con 0x7ff3ec09dfd0 auth_method 0 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3f152a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.518+0000 7ff3e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 10 monclient(hunting): handle_auth_done global_id 84170 payload 824 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 -- 172.21.15.5:0/1586624742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 msgr2=0x7ff3ec1cabf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:01:58.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 0x7ff3ec1cabf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 -- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ec09dfd0 msgr2=0x7ff3ec1cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ec09dfd0 0x7ff3ec1cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 10 monclient: _finish_hunting 0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 monclient: found mon.noname-c 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 -- 172.21.15.5:0/1586624742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc0094b0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 client.?.objecter ms_handle_connect 0x7ff3ec09e8e0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f152a700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400b680 tx=0x7ff3e4004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4004450 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: dump: 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: _finish_auth 0 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: _check_auth_tickets 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 5 monclient: authenticate success, global_id 84170 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e40092e0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _renew_subs 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e4009b30 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4004830 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:01:58.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 monclient: dump: 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 client.84170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _renew_subs 2024-09-16T11:01:58.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec09fab0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 1 librados: init done 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e2ffd700 4 set_mon_vals no callback set 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff3e401c440 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec0a3410 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 --2- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 0x7ff3d80393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(484..484 src has 344..484) v4 ==== 5556+0+0 (secure 0 0 0) 0x7ff3e4049510 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 10 client.84170.objecter ms_dispatch 0x7ff3ec000fd0 osd_map(484..484 src has 344..484) v4 2024-09-16T11:01:58.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 3 client.84170.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 3 client.84170.objecter handle_osd_map decoding full epoch 484 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.522+0000 7ff3e17fa700 20 client.84170.objecter dump_active .. 0 homeless 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.526+0000 7ff3f0d29700 1 --2- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 0x7ff3d80393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.526+0000 7ff3f0d29700 10 monclient: get_auth_request con 0x7ff3d8036f10 auth_method 0 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.526+0000 7ff3f0d29700 1 --2- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 0x7ff3d80393d0 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80039e0 tx=0x7ff3e8009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.526+0000 7ff3e17fa700 10 client.84170.objecter ms_handle_connect 0x7ff3d8036f10 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.530+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff3e40249a0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.530+0000 7ff3e17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:01:58.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.530+0000 7ff3e17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:01:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3f1d2b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3f1d2b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3f1d2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff3ec1336c0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3e17fa700 1 -- 172.21.15.5:0/1586624742 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ff3e40243b0 con 0x7ff3ec09e8e0 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3e17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:01:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.766+0000 7ff3e17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:01:58.773 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 10 librados: watch_flush enter 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 10 librados: watch_flush exit 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 20 client.84170.objecter shutdown clearing up homeless session... 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 10 client.84170.objecter successfully canceled tick 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 msgr2=0x7ff3d80393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 0x7ff3d80393d0 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80039e0 tx=0x7ff3e8009270 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 10 monclient: shutdown 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 msgr2=0x7ff3ec09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400b680 tx=0x7ff3e4004af0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 shutdown_connections 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff3ec09e8e0 0x7ff3ec09b3e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3d8036f10 0x7ff3d80393d0 unknown :-1 s=CLOSED pgs=3853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3ec09dfd0 0x7ff3ec1cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 --2- 172.21.15.5:0/1586624742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3ec09d6a0 0x7ff3ec1cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 >> 172.21.15.5:0/1586624742 conn(0x7ff3ec0a45d0 msgr2=0x7ff3ec09ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 shutdown_connections 2024-09-16T11:01:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 -- 172.21.15.5:0/1586624742 wait complete. 2024-09-16T11:01:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:01:58.770+0000 7ff3f1d2b700 1 librados: shutdown 2024-09-16T11:02:01.787 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:02:01.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: get_monmap_and_config 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: build_initial_monmap 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: monmap: 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:02:01.905805+0000 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:02:01.905805+0000 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:02:01.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:02:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:02:01.908 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:02:01.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 Processor -- start 2024-09-16T11:02:01.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 -- start start 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: init 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f921409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): picked mon.noname-b con 0x7f921409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f921409f200 0x7f9214139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): picked mon.noname-a con 0x7f921409f200 addr v2:172.21.15.5:3300/0 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f921413a460 addr v1:172.21.15.143:6789/0 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921409fb80 con 0x7f921413a460 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:01.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921409c030 con 0x7f921409f200 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921409b970 con 0x7f921409e8e0 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient(hunting): authenticate will time out at 1580.482185s 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f921409f200 0x7f9214139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219a81700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f921413a460 legacy=0x7f92141411e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f921409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f921409f200 0x7f9214139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39748/0 (socket says 172.21.15.5:39748) 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 -- 172.21.15.5:0/558477312 learned_addr learned my addr 172.21.15.5:0/558477312 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): get_auth_request con 0x7f921409f200 auth_method 0 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:01.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): get_auth_request con 0x7f921409e8e0 auth_method 0 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:01.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient(hunting): handle_auth_done global_id 74325 payload 274 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 -- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 msgr2=0x7f921409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f921409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 -- 172.21.15.5:0/558477312 >> v1:172.21.15.143:6789/0 conn(0x7f921413a460 legacy=0x7f92141411e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient: _finish_hunting 0 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 monclient: found mon.noname-a 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 -- 172.21.15.5:0/558477312 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921409bc50 con 0x7f921409f200 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9219280700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f921409ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9218a7f700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921409f200 0x7f9214139f20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92140a13c0 tx=0x7f9204009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d83178afcf24549 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 1 -- 172.21.15.5:0/558477312 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204011830 con 0x7f921409f200 2024-09-16T11:02:01.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: dump: 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:01.916 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: _finish_auth 0 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: _check_auth_tickets 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 5 monclient: authenticate success, global_id 74325 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 1 -- 172.21.15.5:0/558477312 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9204011990 con 0x7f921409f200 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 10 monclient: get_monmap_and_config success 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f921a282700 4 set_mon_vals no callback set 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 1 -- 172.21.15.5:0/558477312 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204011d20 con 0x7f921409f200 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:01.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.902+0000 7f9213fff700 10 monclient: dump: 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 10 monclient: shutdown 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921409f200 msgr2=0x7f9214139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921409f200 0x7f9214139f20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92140a13c0 tx=0x7f9204009bf0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- 172.21.15.5:0/558477312 shutdown_connections 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921409f200 0x7f9214139f20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 --2- 172.21.15.5:0/558477312 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f921409ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- 172.21.15.5:0/558477312 >> 172.21.15.5:0/558477312 conn(0x7f92140a45d0 msgr2=0x7f92140a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- 172.21.15.5:0/558477312 shutdown_connections 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- 172.21.15.5:0/558477312 wait complete. 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 10 monclient: build_initial_monmap 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 10 monclient: monmap: 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:01.919 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:02:01.920 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 librados: starting msgr at 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 librados: starting objecter 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 Processor -- start 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 -- start start 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 librados: setting wanted keys 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 1 librados: calling monclient init 2024-09-16T11:02:01.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.906+0000 7f921a282700 10 monclient: init 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): picked mon.noname-c con 0x7f921409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 0x7f92141cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): picked mon.noname-b con 0x7f921409f200 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921413a460 0x7f92141266a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): picked mon.noname-a con 0x7f921413a460 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921412a010 con 0x7f921413a460 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:01.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921409b0d0 con 0x7f921409f200 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9214139f40 con 0x7f921409e8e0 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient(hunting): authenticate will time out at 1580.488619s 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9218a7f700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 msgr2=0x7f92141cf760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9218a7f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 0x7f92141cf760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:33644/0 (socket says 172.21.15.5:33644) 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 -- 172.21.15.5:0/1882718178 learned_addr learned my addr 172.21.15.5:0/1882718178 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): get_auth_request con 0x7f921409e8e0 auth_method 0 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:01.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921413a460 0x7f92141266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): get_auth_request con 0x7f921413a460 auth_method 0 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219a81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient(hunting): handle_auth_done global_id 84176 payload 824 2024-09-16T11:02:01.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 -- 172.21.15.5:0/1882718178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 msgr2=0x7f92141cf760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 0x7f92141cf760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 -- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921413a460 msgr2=0x7f92141266a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921413a460 0x7f92141266a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient: _finish_hunting 0 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 monclient: found mon.noname-c 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 -- 172.21.15.5:0/1882718178 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92080094b0 con 0x7f921409e8e0 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 client.?.objecter ms_handle_connect 0x7f921409e8e0 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9219280700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f920800ebe0 tx=0x7f920800eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:02:01.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920800c890 con 0x7f921409e8e0 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: dump: 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:01.930 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: _finish_auth 0 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920800c9f0 con 0x7f921409e8e0 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 5 monclient: authenticate success, global_id 84176 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _renew_subs 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:01.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9204009570 con 0x7f921409e8e0 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920800cd80 con 0x7f921409e8e0 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 10 monclient: dump: 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:01.932 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f92137fe700 4 set_mon_vals no callback set 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 client.84176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _renew_subs 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921409a620 con 0x7f921409e8e0 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f921a282700 1 librados: init done 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9208013470 con 0x7f921409e8e0 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.910+0000 7f9211ffb700 1 --2- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 0x7f92000393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:01.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(484..484 src has 344..484) v4 ==== 5556+0+0 (secure 0 0 0) 0x7f9208048390 con 0x7f921409e8e0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 10 client.84176.objecter ms_dispatch 0x7f9214000fd0 osd_map(484..484 src has 344..484) v4 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 3 client.84176.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 3 client.84176.objecter handle_osd_map decoding full epoch 484 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 20 client.84176.objecter dump_active .. 0 homeless 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f921a282700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f921a282700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f921a282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921406fa50 con 0x7f921409e8e0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9218a7f700 1 --2- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 0x7f92000393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9218a7f700 10 monclient: get_auth_request con 0x7f9200036f10 auth_method 0 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9211ffb700 10 client.84176.objecter ms_handle_connect 0x7f9200036f10 2024-09-16T11:02:01.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.914+0000 7f9218a7f700 1 --2- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 0x7f92000393d0 secure :-1 s=READY pgs=3854 cs=0 l=1 rev1=1 crypto rx=0x7f921409dd60 tx=0x7f9204008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.926+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f920801b3b0 con 0x7f921409e8e0 2024-09-16T11:02:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.926+0000 7f9211ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:01.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:01.926+0000 7f9211ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:02:02.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:02:02.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:02.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:02.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f92140a0be0 con 0x7f921409e8e0 2024-09-16T11:02:02.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f9211ffb700 1 -- 172.21.15.5:0/1882718178 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f920803b610 con 0x7f921409e8e0 2024-09-16T11:02:02.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f9211ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:02.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f9211ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:02:02.168 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 librados: watch_flush enter 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 librados: watch_flush exit 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 20 client.84176.objecter shutdown clearing up homeless session... 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 client.84176.objecter successfully canceled tick 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 msgr2=0x7f92000393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 0x7f92000393d0 secure :-1 s=READY pgs=3854 cs=0 l=1 rev1=1 crypto rx=0x7f921409dd60 tx=0x7f9204008030 comp rx=0 tx=0).stop 2024-09-16T11:02:02.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 10 monclient: shutdown 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 msgr2=0x7f92141cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f920800ebe0 tx=0x7f920800eef0 comp rx=0 tx=0).stop 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 shutdown_connections 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9200036f10 0x7f92000393d0 unknown :-1 s=CLOSED pgs=3854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f921413a460 0x7f92141266a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f921409f200 0x7f92141cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 --2- 172.21.15.5:0/1882718178 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f921409e8e0 0x7f92141cf220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 >> 172.21.15.5:0/1882718178 conn(0x7f92140a45d0 msgr2=0x7f9214070200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 shutdown_connections 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.162+0000 7f921a282700 1 -- 172.21.15.5:0/1882718178 wait complete. 2024-09-16T11:02:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:02.166+0000 7f921a282700 1 librados: shutdown 2024-09-16T11:02:05.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: get_monmap_and_config 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: build_initial_monmap 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: monmap: 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:02:05.299079+0000 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:02:05.299079+0000 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:02:05.300 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:02:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:02:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:02:05.301 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:02:05.301 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 Processor -- start 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 -- start start 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: init 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-b con 0x7fc3a009e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3a009d6a0 0x7fc3a009f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-a con 0x7fc3a009d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3a009dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a0070e40 con 0x7fc3a009dfd0 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a00a01d0 con 0x7fc3a009d6a0 2024-09-16T11:02:05.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a009f740 con 0x7fc3a009e8e0 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc3a55dd700 10 monclient(hunting): authenticate will time out at 1583.875472s 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3a009d6a0 0x7fc3a009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3a009d6a0 0x7fc3a009f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45610/0 (socket says 172.21.15.5:45610) 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 1 -- 172.21.15.5:0/1504250137 learned_addr learned my addr 172.21.15.5:0/1504250137 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): get_auth_request con 0x7fc3a009d6a0 auth_method 0 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc39ffff700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:05.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc39ffff700 10 monclient(hunting): get_auth_request con 0x7fc3a009e8e0 auth_method 0 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc39ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc39ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc39ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 10 monclient(hunting): handle_auth_done global_id 74331 payload 274 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 1 -- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 msgr2=0x7fc3a009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.294+0000 7fc397fff700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a009ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 1 -- 172.21.15.5:0/1504250137 >> v1:172.21.15.5:6789/0 conn(0x7fc3a009dfd0 legacy=0x7fc3a01336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 10 monclient: _finish_hunting 0 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 1 monclient: found mon.noname-a 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 1 -- 172.21.15.5:0/1504250137 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a009fef0 con 0x7fc3a009d6a0 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc397fff700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009d6a0 0x7fc3a009f200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc388009910 tx=0x7fc388009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d011aefb356981 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 1 -- 172.21.15.5:0/1504250137 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388004230 con 0x7fc3a009d6a0 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: dump: 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 5 monclient: authenticate success, global_id 74331 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 1 -- 172.21.15.5:0/1504250137 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388004390 con 0x7fc3a009d6a0 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 10 monclient: get_monmap_and_config success 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 4 set_mon_vals no callback set 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 1 -- 172.21.15.5:0/1504250137 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388009e60 con 0x7fc3a009d6a0 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:05.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc39f7fe700 10 monclient: dump: 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 10 monclient: shutdown 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:05.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 -- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009d6a0 msgr2=0x7fc3a009f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009d6a0 0x7fc3a009f200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc388009910 tx=0x7fc388009c20 comp rx=0 tx=0).stop 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 -- 172.21.15.5:0/1504250137 shutdown_connections 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009d6a0 0x7fc3a009f200 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/1504250137 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 -- 172.21.15.5:0/1504250137 >> 172.21.15.5:0/1504250137 conn(0x7fc3a00a45d0 msgr2=0x7fc3a00a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 -- 172.21.15.5:0/1504250137 shutdown_connections 2024-09-16T11:02:05.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 -- 172.21.15.5:0/1504250137 wait complete. 2024-09-16T11:02:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 10 monclient: build_initial_monmap 2024-09-16T11:02:05.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 10 monclient: monmap: 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 librados: starting msgr at 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.298+0000 7fc3a55dd700 1 librados: starting objecter 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 Processor -- start 2024-09-16T11:02:05.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- start start 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 librados: setting wanted keys 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 librados: calling monclient init 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: init 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 0x7fc3a01c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-b con 0x7fc3a009d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-a con 0x7fc3a009dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a01331a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): picked mon.noname-c con 0x7fc3a009e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:05.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a011fd80 con 0x7fc3a009dfd0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a009b0d0 con 0x7fc3a009d6a0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00a3980 con 0x7fc3a009e8e0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient(hunting): authenticate will time out at 1583.881527s 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39ffff700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 msgr2=0x7fc3a01c8390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 0x7fc3a01c8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45624/0 (socket says 172.21.15.5:45624) 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 -- 172.21.15.5:0/3975708123 learned_addr learned my addr 172.21.15.5:0/3975708123 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): get_auth_request con 0x7fc3a009dfd0 auth_method 0 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:05.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a01331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 10 monclient(hunting): get_auth_request con 0x7fc3a009e8e0 auth_method 0 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient(hunting): handle_auth_done global_id 74334 payload 824 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 -- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 msgr2=0x7fc3a01331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a01331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 -- 172.21.15.5:0/3975708123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 msgr2=0x7fc3a01c8390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 0x7fc3a01c8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient: _finish_hunting 0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 monclient: found mon.noname-a 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 -- 172.21.15.5:0/3975708123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388009570 con 0x7fc3a009dfd0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a4ddc700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a01331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 client.?.objecter ms_handle_connect 0x7fc3a009dfd0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc397fff700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00a0940 tx=0x7fc388004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388009e80 con 0x7fc3a009dfd0 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: dump: 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:05.319 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: _finish_auth 0 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: _check_auth_tickets 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 5 monclient: authenticate success, global_id 74334 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3880036c0 con 0x7fc3a009dfd0 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:05.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _renew_subs 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388003a50 con 0x7fc3a009dfd0 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a0120620 con 0x7fc3a009dfd0 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39d7fa700 10 monclient: dump: 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:05.321 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc39effd700 4 set_mon_vals no callback set 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 client.74334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _renew_subs 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a011e2c0 con 0x7fc3a009dfd0 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.302+0000 7fc3a55dd700 1 librados: init done 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc388021070 con 0x7fc3a009dfd0 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc3a55dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc3a55dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc3a55dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a01341b0 con 0x7fc3a009dfd0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 1 --2- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 0x7fc3800393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(484..484 src has 344..484) v4 ==== 5556+0+0 (secure 0 0 0) 0x7fc388049980 con 0x7fc3a009dfd0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 10 client.74334.objecter ms_dispatch 0x7fc3a0000fd0 osd_map(484..484 src has 344..484) v4 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 3 client.74334.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 3 client.74334.objecter handle_osd_map decoding full epoch 484 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 20 client.74334.objecter dump_active .. 0 homeless 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39ffff700 1 --2- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 0x7fc3800393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39ffff700 10 monclient: get_auth_request con 0x7fc380036f10 auth_method 0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39d7fa700 10 client.74334.objecter ms_handle_connect 0x7fc380036f10 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.306+0000 7fc39ffff700 1 --2- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 0x7fc3800393d0 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7fc390009100 tx=0x7fc390008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:05.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.310+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc388024470 con 0x7fc3a009dfd0 2024-09-16T11:02:05.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.310+0000 7fc39d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:05.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.310+0000 7fc39d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:02:05.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.542+0000 7fc3a55dd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:02:05.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.542+0000 7fc3a55dd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:05.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.542+0000 7fc3a55dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:05.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.542+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc3a01336e0 con 0x7fc3a009dfd0 2024-09-16T11:02:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.546+0000 7fc39d7fa700 1 -- 172.21.15.5:0/3975708123 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fc38801abd0 con 0x7fc3a009dfd0 2024-09-16T11:02:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.546+0000 7fc39d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.546+0000 7fc39d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:02:05.550 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:02:05.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 10 librados: watch_flush enter 2024-09-16T11:02:05.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 10 librados: watch_flush exit 2024-09-16T11:02:05.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 20 client.74334.objecter shutdown clearing up homeless session... 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 10 client.74334.objecter successfully canceled tick 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 msgr2=0x7fc3800393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 0x7fc3800393d0 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7fc390009100 tx=0x7fc390008040 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 10 monclient: shutdown 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 msgr2=0x7fc3a0132c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00a0940 tx=0x7fc388004770 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 shutdown_connections 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc3a009e8e0 0x7fc3a01331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc380036f10 0x7fc3800393d0 unknown :-1 s=CLOSED pgs=3855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3a009dfd0 0x7fc3a0132c60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 --2- 172.21.15.5:0/3975708123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3a009d6a0 0x7fc3a01c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 >> 172.21.15.5:0/3975708123 conn(0x7fc3a00a45d0 msgr2=0x7fc3a01bbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:05.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 shutdown_connections 2024-09-16T11:02:05.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 -- 172.21.15.5:0/3975708123 wait complete. 2024-09-16T11:02:05.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:05.550+0000 7fc3a55dd700 1 librados: shutdown 2024-09-16T11:02:08.572 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:02:08.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: get_monmap_and_config 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: build_initial_monmap 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: monmap: 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:02:08.689683+0000 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:02:08.689683+0000 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:02:08.691 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:02:08.692 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 Processor -- start 2024-09-16T11:02:08.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 -- start start 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: init 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-b con 0x7f398409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-a con 0x7f398409d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-c con 0x7f398409dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840a01d0 con 0x7f398409d6a0 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:08.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3984070e40 con 0x7f398409dfd0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398409f740 con 0x7f398409e8e0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3989eeb700 10 monclient(hunting): authenticate will time out at 1587.266045s 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f39896ea700 1 -- >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 msgr2=0x7f39841336c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f39896ea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:50118/0 (socket says 172.21.15.5:50118) 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 learned_addr learned my addr 172.21.15.5:0/2379539700 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f398409e8e0 auth_method 0 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:08.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 10 monclient(hunting): get_auth_request con 0x7f398409d6a0 auth_method 0 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:08.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.686+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:08.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f39896ea700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 msgr2=0x7f39841336c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f39896ea700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 10 monclient(hunting): get_auth_request con 0x7f398409d6a0 auth_method 0 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f398409e8e0 auth_method 0 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:08.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:08.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:08.890+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f39896ea700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 msgr2=0x7f39841336c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f39896ea700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 10 monclient(hunting): get_auth_request con 0x7f398409d6a0 auth_method 0 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:02:09.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f398409e8e0 auth_method 0 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:09.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:09.294+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f39896ea700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 msgr2=0x7f39841336c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f39896ea700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 10 monclient(hunting): get_auth_request con 0x7f398409d6a0 auth_method 0 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:10.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f398409e8e0 auth_method 0 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:02:10.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:10.098+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:02:11.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): tick 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 monclient(hunting): continuing hunt 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 msgr2=0x7f398409ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f398409e8e0 0x7f398409ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 msgr2=0x7f39841336c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.143:3300/0 conn(0x7f398409dfd0 0x7f39841336c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 -- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 msgr2=0x7f398409f200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 --2- 172.21.15.5:0/2379539700 >> v2:172.21.15.5:3300/0 conn(0x7f398409d6a0 0x7f398409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient: _add_conns ranks=[1,0] 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3964003710 addr v1:172.21.15.143:6789/0 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3964007440 addr v1:172.21.15.5:6789/0 2024-09-16T11:02:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 -- 172.21.15.5:0/2379539700 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398409fef0 con 0x7f3964007440 2024-09-16T11:02:11.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 1 -- 172.21.15.5:0/2379539700 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396400c1c0 con 0x7f3964003710 2024-09-16T11:02:11.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f39827fc700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:11.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.690+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.143:6789/0 conn(0x7f3964003710 legacy=0x7f3964005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:11.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:11.894+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.143:6789/0 conn(0x7f3964003710 legacy=0x7f3964005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:12.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:12.298+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.143:6789/0 conn(0x7f3964003710 legacy=0x7f3964005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:13.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.102+0000 7f3988ee9700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.143:6789/0 conn(0x7f3964003710 legacy=0x7f3964005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478469460 0 0) 0x7f398409fef0 con 0x7f3964007440 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 10 monclient(hunting): _negotiate 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 10 monclient(hunting): my global_id is 74337 2024-09-16T11:02:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.166+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39700035d0 con 0x7f3964007440 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223558543 0 0) 0x7f39700035d0 con 0x7f3964007440 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.143:6789/0 conn(0x7f3964003710 legacy=0x7f3964005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 monclient: found mon.noname-a-legacy 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396400d3c0 con 0x7f3964007440 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f396c009980 con 0x7f3964007440 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3989eeb700 5 monclient: authenticate success, global_id 74337 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3989eeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: dump: 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 >> v1:172.21.15.5:6789/0 conn(0x7f3964007440 legacy=0x7f3964009900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39896ea700 1 -- 172.21.15.5:0/2379539700 reap_dead start 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 0x7f3970003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): picked mon.a con 0x7f3970003a50 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 0x7f3970004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): picked mon.c con 0x7f3970004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 0x7f397000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): picked mon.b con 0x7f3970005350 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397000c240 con 0x7f3970003a50 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397000c3b0 con 0x7f3970005350 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397000c570 con 0x7f3970004870 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39837fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3989eeb700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 0x7f3970003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39896ea700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 msgr2=0x7f397000ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f39896ea700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 0x7f397000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f3970003a50 auth_method 0 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 0x7f3970004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): get_auth_request con 0x7f3970004870 auth_method 0 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient(hunting): handle_auth_done global_id 74337 payload 293 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 msgr2=0x7f397000ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 0x7f397000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 msgr2=0x7f3970003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 0x7f3970003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 monclient: found mon.c 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 1 -- 172.21.15.5:0/2379539700 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700035d0 con 0x7f3970004870 2024-09-16T11:02:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3983fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.206+0000 7f3988ee9700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 0x7f3970003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:02:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3983fff700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 0x7f3970004cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f396c007090 tx=0x7f39700058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1a3c8db69445ae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c018790 con 0x7f3970004870 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: dump: 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.225 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c0188f0 con 0x7f3970004870 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 10 monclient: get_monmap_and_config success 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 4 set_mon_vals no callback set 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 1 -- 172.21.15.5:0/2379539700 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c018c80 con 0x7f3970004870 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f39837fe700 10 monclient: dump: 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.226 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 10 monclient: shutdown 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 1 -- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 msgr2=0x7f3970004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 0x7f3970004cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f396c007090 tx=0x7f39700058a0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.210+0000 7f3989eeb700 1 -- 172.21.15.5:0/2379539700 shutdown_connections 2024-09-16T11:02:13.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970005350 0x7f397000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970004870 0x7f3970004cf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- 172.21.15.5:0/2379539700 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970003a50 0x7f3970003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- 172.21.15.5:0/2379539700 >> 172.21.15.5:0/2379539700 conn(0x7f39840a45d0 msgr2=0x7f39840a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- 172.21.15.5:0/2379539700 shutdown_connections 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- 172.21.15.5:0/2379539700 wait complete. 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient: build_initial_monmap 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient: monmap: 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:13.228 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 librados: starting msgr at 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 librados: starting objecter 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 Processor -- start 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- start start 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 librados: setting wanted keys 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 librados: calling monclient init 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient: init 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-c con 0x7f3970003a50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 0x7f39841337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-b con 0x7f3970004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970005350 0x7f3984133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): picked mon.noname-a con 0x7f3970005350 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39841c5f50 con 0x7f3970005350 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840a31e0 con 0x7f3970004870 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398409d1e0 con 0x7f3970003a50 2024-09-16T11:02:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3989eeb700 10 monclient(hunting): authenticate will time out at 1591.795604s 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3983fff700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 msgr2=0x7f39841337f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3983fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 0x7f39841337f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52528/0 (socket says 172.21.15.5:52528) 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 1 -- 172.21.15.5:0/870021768 learned_addr learned my addr 172.21.15.5:0/870021768 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 10 monclient(hunting): get_auth_request con 0x7f3970003a50 auth_method 0 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f3988ee9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f39896ea700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970005350 0x7f3984133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f39896ea700 10 monclient(hunting): get_auth_request con 0x7f3970005350 auth_method 0 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f39896ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f39896ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.214+0000 7f39896ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39896ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39896ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39896ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient(hunting): handle_auth_done global_id 84185 payload 824 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 -- 172.21.15.5:0/870021768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 msgr2=0x7f39841337f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 0x7f39841337f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 -- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970005350 msgr2=0x7f3984133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970005350 0x7f3984133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 monclient: found mon.noname-c 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 -- 172.21.15.5:0/870021768 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398409b0d0 con 0x7f3970003a50 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3988ee9700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3974001eb0 tx=0x7f3974017ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 client.?.objecter ms_handle_connect 0x7f3970003a50 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974018770 con 0x7f3970003a50 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: dump: 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 5 monclient: authenticate success, global_id 84185 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397401c070 con 0x7f3970003a50 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974018b10 con 0x7f3970003a50 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _renew_subs 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39841221c0 con 0x7f3970003a50 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3982ffd700 4 set_mon_vals no callback set 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 client.84185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 monclient: dump: 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _renew_subs 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398411fe60 con 0x7f3970003a50 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 1 librados: init done 2024-09-16T11:02:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f397401e070 con 0x7f3970003a50 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 --2- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 0x7f3964039be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(485..485 src has 344..485) v4 ==== 5666+0+0 (secure 0 0 0) 0x7f3974048990 con 0x7f3970003a50 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 client.84185.objecter ms_dispatch 0x7f3984000fd0 osd_map(485..485 src has 344..485) v4 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 3 client.84185.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 3 client.84185.objecter handle_osd_map decoding full epoch 485 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 20 client.84185.objecter dump_active .. 0 homeless 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398409ee70 con 0x7f3970003a50 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3983fff700 1 --2- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 0x7f3964039be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3983fff700 10 monclient: get_auth_request con 0x7f3964037720 auth_method 0 2024-09-16T11:02:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f3983fff700 1 --2- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 0x7f3964039be0 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7f396c007090 tx=0x7f396c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.218+0000 7f39817fa700 10 client.84185.objecter ms_handle_connect 0x7f3964037720 2024-09-16T11:02:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.226+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3974029350 con 0x7f3970003a50 2024-09-16T11:02:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.226+0000 7f39817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.226+0000 7f39817fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:02:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.458+0000 7f3989eeb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:02:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.458+0000 7f3989eeb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:13.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.458+0000 7f3989eeb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.458+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3984120c50 con 0x7f3970003a50 2024-09-16T11:02:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.462+0000 7f39817fa700 1 -- 172.21.15.5:0/870021768 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f397401fe90 con 0x7f3970003a50 2024-09-16T11:02:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.462+0000 7f39817fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:02:13.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.462+0000 7f39817fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:02:13.466 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:02:13.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.462+0000 7f3989eeb700 10 librados: watch_flush enter 2024-09-16T11:02:13.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 10 librados: watch_flush exit 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 20 client.84185.objecter shutdown clearing up homeless session... 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 10 client.84185.objecter successfully canceled tick 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 msgr2=0x7f3964039be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 0x7f3964039be0 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7f396c007090 tx=0x7f396c011000 comp rx=0 tx=0).stop 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 10 monclient: shutdown 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 msgr2=0x7f39841c5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3974001eb0 tx=0x7f3974017ab0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 shutdown_connections 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3964037720 0x7f3964039be0 unknown :-1 s=CLOSED pgs=3856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3970005350 0x7f3984133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3970004870 0x7f39841337f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 --2- 172.21.15.5:0/870021768 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3970003a50 0x7f39841c5a10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 >> 172.21.15.5:0/870021768 conn(0x7f39840a45d0 msgr2=0x7f39840a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 shutdown_connections 2024-09-16T11:02:13.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 -- 172.21.15.5:0/870021768 wait complete. 2024-09-16T11:02:13.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.466+0000 7f3989eeb700 1 librados: shutdown 2024-09-16T11:02:13.487 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T11:02:13.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T11:02:13.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:02:13.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: get_monmap_and_config 2024-09-16T11:02:13.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: build_initial_monmap 2024-09-16T11:02:13.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: monmap: 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:02:13.601328+0000 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:02:13.601328+0000 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:02:13.603 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 Processor -- start 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 -- start start 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: init 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba080a2bf0 addr v1:172.21.15.143:6789/0 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 0x7fba0813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-b con 0x7fba0809f780 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba080a36e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba080a3c40 con 0x7fba080a36e0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba080a4300 con 0x7fba080a2bf0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0809fda0 con 0x7fba0809f780 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba07fff700 10 monclient(hunting): authenticate will time out at 1592.178422s 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba06ffd700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fba080a2bf0 legacy=0x7fba080a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba077fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fba080a36e0 0x7fba0813d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43278/0 (socket says 172.21.15.5:43278) 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba077fe700 1 -- 172.21.15.5:0/461315370 learned_addr learned my addr 172.21.15.5:0/461315370 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 --2- 172.21.15.5:0/461315370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 0x7fba0813ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): get_auth_request con 0x7fba0809f780 auth_method 0 2024-09-16T11:02:13.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 1 -- 172.21.15.5:0/461315370 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020590785 0 0) 0x7fba080a3c40 con 0x7fba080a36e0 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 10 monclient(hunting): _negotiate 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 10 monclient(hunting): my global_id is 74343 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba05ffb700 1 -- 172.21.15.5:0/461315370 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f40035d0 con 0x7fba080a36e0 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient(hunting): handle_auth_done global_id 84188 payload 274 2024-09-16T11:02:13.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 -- 172.21.15.5:0/461315370 >> v1:172.21.15.143:6789/0 conn(0x7fba080a2bf0 legacy=0x7fba080a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 -- 172.21.15.5:0/461315370 >> v1:172.21.15.5:6789/0 conn(0x7fba080a36e0 legacy=0x7fba0813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 monclient: found mon.noname-b 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 -- 172.21.15.5:0/461315370 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080a0080 con 0x7fba0809f780 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.598+0000 7fba067fc700 1 --2- 172.21.15.5:0/461315370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 0x7fba0813ae00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009880 tx=0x7fb9f8009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43333463a09d5aca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 1 -- 172.21.15.5:0/461315370 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8004230 con 0x7fba0809f780 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: dump: 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.611 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 1 -- 172.21.15.5:0/461315370 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8004390 con 0x7fba0809f780 2024-09-16T11:02:13.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 5 monclient: authenticate success, global_id 84188 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 1 -- 172.21.15.5:0/461315370 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80036c0 con 0x7fba0809f780 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 4 set_mon_vals no callback set 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba05ffb700 10 monclient: dump: 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.613 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 10 monclient: shutdown 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 -- 172.21.15.5:0/461315370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 msgr2=0x7fba0813ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 --2- 172.21.15.5:0/461315370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 0x7fba0813ae00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009880 tx=0x7fb9f8009b90 comp rx=0 tx=0).stop 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 -- 172.21.15.5:0/461315370 shutdown_connections 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 --2- 172.21.15.5:0/461315370 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba0809f780 0x7fba0813ae00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 -- 172.21.15.5:0/461315370 >> 172.21.15.5:0/461315370 conn(0x7fba080a53e0 msgr2=0x7fba08073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 -- 172.21.15.5:0/461315370 shutdown_connections 2024-09-16T11:02:13.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 -- 172.21.15.5:0/461315370 wait complete. 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 10 monclient: build_initial_monmap 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 10 monclient: monmap: 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:13.621 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 librados: starting msgr at 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 librados: starting objecter 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.602+0000 7fba07fff700 1 Processor -- start 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- start start 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 librados: setting wanted keys 2024-09-16T11:02:13.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 librados: calling monclient init 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: init 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-a con 0x7fba0809f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba080a2bf0 0x7fba0813aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-c con 0x7fba080a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba080a36e0 0x7fba0813b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): picked mon.noname-b con 0x7fba080a36e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0813df70 con 0x7fba0809f780 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080a7800 con 0x7fba080a36e0 2024-09-16T11:02:13.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0811ed30 con 0x7fba080a2bf0 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): authenticate will time out at 1592.185264s 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38514/0 (socket says 172.21.15.5:38514) 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 -- 172.21.15.5:0/4207998029 learned_addr learned my addr 172.21.15.5:0/4207998029 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): get_auth_request con 0x7fba0809f780 auth_method 0 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba077fe700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba080a36e0 msgr2=0x7fba0813b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:13.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba080a2bf0 0x7fba0813aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba077fe700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba080a36e0 0x7fba0813b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 10 monclient(hunting): get_auth_request con 0x7fba080a2bf0 auth_method 0 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient(hunting): handle_auth_done global_id 74346 payload 824 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba080a2bf0 msgr2=0x7fba0813aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba080a2bf0 0x7fba0813aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba080a36e0 msgr2=0x7fba0813b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:02:13.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba080a36e0 0x7fba0813b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 monclient: found mon.noname-a 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80094b0 con 0x7fba0809f780 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fba080a2bf0 0x7fba0813aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba06ffd700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000b780 tx=0x7fb9f000bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 client.?.objecter ms_handle_connect 0x7fba0809f780 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:02:13.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0003e70 con 0x7fba0809f780 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: dump: 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.629 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 5 monclient: authenticate success, global_id 74346 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0005070 con 0x7fba0809f780 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0005420 con 0x7fba0809f780 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _renew_subs 2024-09-16T11:02:13.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0813cfb0 con 0x7fba0809f780 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba057fa700 4 set_mon_vals no callback set 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 10 monclient: dump: 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.631 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 client.74346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _renew_subs 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba08136940 con 0x7fba0809f780 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 librados: init done 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb9f0005580 con 0x7fba0809f780 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 0x7fba081371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.606+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08074d50 con 0x7fba08136de0 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 0x7fba081371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 --2- 172.21.15.5:0/4207998029 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9e8036f10 0x7fb9e80393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient: get_auth_request con 0x7fba08136de0 auth_method 0 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(485..485 src has 344..485) v4 ==== 5666+0+0 (secure 0 0 0) 0x7fb9f00479b0 con 0x7fba0809f780 2024-09-16T11:02:13.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 client.74346.objecter ms_dispatch 0x7fba08000fd0 osd_map(485..485 src has 344..485) v4 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 3 client.74346.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 3 client.74346.objecter handle_osd_map decoding full epoch 485 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 10 monclient(hunting): handle_auth_done global_id 74349 payload 824 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0813ea70 con 0x7fba08136de0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 20 client.74346.objecter dump_active .. 0 homeless 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba077fe700 1 --2- 172.21.15.5:0/4207998029 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9e8036f10 0x7fb9e80393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 0x7fba081371c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009fd0 tx=0x7fb9f8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba077fe700 10 monclient: get_auth_request con 0x7fb9e8036f10 auth_method 0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 client.74346.objecter ms_handle_connect 0x7fba08136de0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 client.74346.objecter resend_mon_ops 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8006ea0 con 0x7fba08136de0 2024-09-16T11:02:13.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 monclient: dump: 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fb9f8020070 con 0x7fba08136de0 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 msgr2=0x7fba081371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 0x7fba081371c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009fd0 tx=0x7fb9f8004770 comp rx=0 tx=0).stop 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 msgr2=0x7fba081371c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08136de0 0x7fba081371c0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8009fd0 tx=0x7fb9f8004770 comp rx=0 tx=0).stop 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fba077fe700 1 --2- 172.21.15.5:0/4207998029 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9e8036f10 0x7fb9e80393d0 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7fba0809f6c0 tx=0x7fba00001710 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.610+0000 7fb9ff7fe700 10 client.74346.objecter ms_handle_connect 0x7fb9e8036f10 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 0x7fba081d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080a2fd0 con 0x7fba08137700 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 0x7fba081d49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient: get_auth_request con 0x7fba08137700 auth_method 0 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 10 monclient(hunting): handle_auth_done global_id 74352 payload 824 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 1 -- 172.21.15.5:0/4207998029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fba080008d0 con 0x7fba08137700 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba067fc700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 0x7fba081d49c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80097d0 tx=0x7fb9f8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 client.74346.objecter ms_handle_connect 0x7fba08137700 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 client.74346.objecter resend_mon_ops 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8011630 con 0x7fba08137700 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 monclient: dump: 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.639 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7fb9f8011790 con 0x7fba08137700 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 msgr2=0x7fba081d49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 0x7fba081d49c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80097d0 tx=0x7fb9f8011000 comp rx=0 tx=0).stop 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 msgr2=0x7fba081d49c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fb9ff7fe700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba08137700 0x7fba081d49c0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80097d0 tx=0x7fb9f8011000 comp rx=0 tx=0).stop 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 52, 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:02:13.641 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.642 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.643 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:02:13.644 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.645 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:02:13.646 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.647 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.648 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.649 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.650 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.651 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:02:13.652 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:02:13.653 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:02:13.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 librados: watch_flush enter 2024-09-16T11:02:13.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 librados: watch_flush exit 2024-09-16T11:02:13.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 20 client.74346.objecter shutdown clearing up homeless session... 2024-09-16T11:02:13.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 client.74346.objecter successfully canceled tick 2024-09-16T11:02:13.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9e8036f10 msgr2=0x7fb9e80393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 --2- 172.21.15.5:0/4207998029 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb9e8036f10 0x7fb9e80393d0 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7fba0809f6c0 tx=0x7fba00001710 comp rx=0 tx=0).stop 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 10 monclient: shutdown 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 msgr2=0x7fba0813a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba077fe700 1 -- 172.21.15.5:0/4207998029 reap_dead start 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000b780 tx=0x7fb9f000bb40 comp rx=0 tx=0).stop 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 shutdown_connections 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 --2- 172.21.15.5:0/4207998029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba0809f780 0x7fba0813a9b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 >> 172.21.15.5:0/4207998029 conn(0x7fba080a53e0 msgr2=0x7fba080a6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.622+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 shutdown_connections 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.626+0000 7fba07fff700 1 -- 172.21.15.5:0/4207998029 wait complete. 2024-09-16T11:02:13.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.626+0000 7fba07fff700 1 librados: shutdown 2024-09-16T11:02:13.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:02:13.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T11:02:13.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:02:13.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.774+0000 7fa149d3f700 10 monclient: get_monmap_and_config 2024-09-16T11:02:13.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.774+0000 7fa149d3f700 10 monclient: build_initial_monmap 2024-09-16T11:02:13.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.774+0000 7fa149d3f700 10 monclient: monmap: 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:02:13.780601+0000 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:02:13.780601+0000 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:02:13.782 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 Processor -- start 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- start start 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient: init 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa14409e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa144138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-b con 0x7fa14409d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa14409dfd0 0x7fa14413b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-a con 0x7fa14409dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa144070e40 con 0x7fa14409e8e0 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1440a01d0 con 0x7fa14409dfd0 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14409fef0 con 0x7fa14409d6a0 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient(hunting): authenticate will time out at 1592.357156s 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa148d3d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa14409e8e0 0x7fa14409ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43290/0 (socket says 172.21.15.5:43290) 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa144138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa148d3d700 1 -- 172.21.15.5:0/3273304311 learned_addr learned my addr 172.21.15.5:0/3273304311 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:02:13.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): get_auth_request con 0x7fa14409d6a0 auth_method 0 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 1 --2- 172.21.15.5:0/3273304311 >> v2:172.21.15.5:3300/0 conn(0x7fa14409dfd0 0x7fa14413b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): get_auth_request con 0x7fa14409dfd0 auth_method 0 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa14953e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient(hunting): handle_auth_done global_id 84194 payload 274 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 -- 172.21.15.5:0/3273304311 >> v2:172.21.15.5:3300/0 conn(0x7fa14409dfd0 msgr2=0x7fa14413b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 --2- 172.21.15.5:0/3273304311 >> v2:172.21.15.5:3300/0 conn(0x7fa14409dfd0 0x7fa14413b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 -- 172.21.15.5:0/3273304311 >> v1:172.21.15.5:6789/0 conn(0x7fa14409e8e0 legacy=0x7fa14409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 monclient: found mon.noname-b 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 -- 172.21.15.5:0/3273304311 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14409f9f0 con 0x7fa14409d6a0 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa143fff700 1 --2- 172.21.15.5:0/3273304311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa144138b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009850 tx=0x7fa12c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83770e28ecab2399 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 1 -- 172.21.15.5:0/3273304311 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c011830 con 0x7fa14409d6a0 2024-09-16T11:02:13.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: dump: 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 5 monclient: authenticate success, global_id 84194 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 1 -- 172.21.15.5:0/3273304311 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c011990 con 0x7fa14409d6a0 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient: get_monmap_and_config success 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 4 set_mon_vals no callback set 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 1 -- 172.21.15.5:0/3273304311 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c011d40 con 0x7fa14409d6a0 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa1437fe700 10 monclient: dump: 2024-09-16T11:02:13.799 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 10 monclient: shutdown 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- 172.21.15.5:0/3273304311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 msgr2=0x7fa144138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 --2- 172.21.15.5:0/3273304311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa144138b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009850 tx=0x7fa12c009b60 comp rx=0 tx=0).stop 2024-09-16T11:02:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- 172.21.15.5:0/3273304311 shutdown_connections 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 --2- 172.21.15.5:0/3273304311 >> v2:172.21.15.5:3300/0 conn(0x7fa14409dfd0 0x7fa14413b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 --2- 172.21.15.5:0/3273304311 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa144138b80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- 172.21.15.5:0/3273304311 >> 172.21.15.5:0/3273304311 conn(0x7fa1440a45d0 msgr2=0x7fa1440a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- 172.21.15.5:0/3273304311 shutdown_connections 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.778+0000 7fa149d3f700 1 -- 172.21.15.5:0/3273304311 wait complete. 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient: build_initial_monmap 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient: monmap: 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:02:13.801 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 librados: starting msgr at 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 librados: starting objecter 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 Processor -- start 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 -- start start 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 librados: setting wanted keys 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 librados: calling monclient init 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient: init 2024-09-16T11:02:13.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient: _reopen_session rank -1 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa1441ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-c con 0x7fa14409d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-a con 0x7fa14409dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14409e8e0 0x7fa1441341a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): picked mon.noname-b con 0x7fa14409e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1441ca8f0 con 0x7fa14409dfd0 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14409b0d0 con 0x7fa14409e8e0 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa144123080 con 0x7fa14409d6a0 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): _renew_subs 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa149d3f700 10 monclient(hunting): authenticate will time out at 1592.363275s 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa1441ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa14953e700 1 -- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14409e8e0 msgr2=0x7fa1441341a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.143:3300/0 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38548/0 (socket says 172.21.15.5:38548) 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa1441ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52570/0 (socket says 172.21.15.5:52570) 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa14953e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14409e8e0 0x7fa1441341a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 1 -- 172.21.15.5:0/1048578439 learned_addr learned my addr 172.21.15.5:0/1048578439 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): get_auth_request con 0x7fa14409d6a0 auth_method 0 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): get_auth_request con 0x7fa14409dfd0 auth_method 0 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient(hunting): handle_auth_done global_id 74361 payload 824 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 msgr2=0x7fa1441ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409d6a0 0x7fa1441ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14409e8e0 msgr2=0x7fa1441341a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa14409e8e0 0x7fa1441341a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient: _finish_hunting 0 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 monclient: found mon.noname-a 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.782+0000 7fa143fff700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c0094b0 con 0x7fa14409dfd0 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa143fff700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa12c016000 tx=0x7fa12c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.?.objecter ms_handle_connect 0x7fa14409dfd0 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0036c0 con 0x7fa14409dfd0 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: dump: 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.807 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: _finish_auth 0 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: _check_auth_tickets 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 5 monclient: authenticate success, global_id 74361 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c003820 con 0x7fa14409dfd0 2024-09-16T11:02:13.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 monclient: _renew_subs 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14406cdb0 con 0x7fa14409dfd0 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c003bb0 con 0x7fa14409dfd0 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa142ffd700 4 set_mon_vals no callback set 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: dump: 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.809 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 client.74361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 monclient: _renew_subs 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa144123640 con 0x7fa14409dfd0 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 1 librados: init done 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 0x7fa14409b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa144067380 con 0x7fa14409b410 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 0x7fa14409b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient: get_auth_request con 0x7fa14409b410 auth_method 0 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fa12c021070 con 0x7fa14409dfd0 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 --2- 172.21.15.5:0/1048578439 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa124036f10 0x7fa1240393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 10 monclient(hunting): handle_auth_done global_id 84200 payload 824 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa14409f3d0 con 0x7fa14409b410 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(485..485 src has 344..485) v4 ==== 5666+0+0 (secure 0 0 0) 0x7fa12c049370 con 0x7fa14409dfd0 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.74361.objecter ms_dispatch 0x7fa144000fd0 osd_map(485..485 src has 344..485) v4 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 3 client.74361.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 3 client.74361.objecter handle_osd_map decoding full epoch 485 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 20 client.74361.objecter dump_active .. 0 homeless 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa148d3d700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 0x7fa14409b7f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa13400fb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.74361.objecter ms_handle_connect 0x7fa14409b410 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.74361.objecter resend_mon_ops 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134005b70 con 0x7fa14409b410 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa14953e700 1 --2- 172.21.15.5:0/1048578439 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa124036f10 0x7fa1240393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: dump: 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa14953e700 10 monclient: get_auth_request con 0x7fa124036f10 auth_method 0 2024-09-16T11:02:13.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fa134005cd0 con 0x7fa14409b410 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 msgr2=0x7fa14409b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 0x7fa14409b7f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa13400fb00 comp rx=0 tx=0).stop 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 msgr2=0x7fa14409b7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409b410 0x7fa14409b7f0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa13400fb00 comp rx=0 tx=0).stop 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa1417fa700 10 client.74361.objecter ms_handle_connect 0x7fa124036f10 2024-09-16T11:02:13.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.786+0000 7fa14953e700 1 --2- 172.21.15.5:0/1048578439 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa124036f10 0x7fa1240393d0 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa14406b9c0 tx=0x7fa13c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 0x7fa1440a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1441234f0 con 0x7fa14409bd30 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 0x7fa1440a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:02:13.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient: get_auth_request con 0x7fa14409bd30 auth_method 0 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 10 monclient(hunting): handle_auth_done global_id 84203 payload 824 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 1 -- 172.21.15.5:0/1048578439 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa14409ecc0 con 0x7fa14409bd30 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 client.74361.objecter ms_handle_connect 0x7fa14409bd30 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa148d3d700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 0x7fa1440a1f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa1340058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 client.74361.objecter resend_mon_ops 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134023070 con 0x7fa14409bd30 2024-09-16T11:02:13.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 monclient: dump: 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:02:13.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4766 (secure 0 0 0) 0x7fa1340076f0 con 0x7fa14409bd30 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 msgr2=0x7fa1440a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 0x7fa1440a1f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa1340058a0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 msgr2=0x7fa1440a1f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa1417fa700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa14409bd30 0x7fa1440a1f20 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa134000f60 tx=0x7fa1340058a0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:02:13.819 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 52, 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.820 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.821 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:02:13.822 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:02:13.823 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:02:13.824 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:02:13.827 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:02:13.828 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:02:13.829 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.829 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:02:13.829 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.832 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.833 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.834 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 7 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:02:13.835 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:02:13.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 librados: watch_flush enter 2024-09-16T11:02:13.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 librados: watch_flush exit 2024-09-16T11:02:13.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 20 client.74361.objecter shutdown clearing up homeless session... 2024-09-16T11:02:13.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 client.74361.objecter successfully canceled tick 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa124036f10 msgr2=0x7fa1240393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 --2- 172.21.15.5:0/1048578439 >> v2:172.21.15.143:6812/1098205034 conn(0x7fa124036f10 0x7fa1240393d0 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa14406b9c0 tx=0x7fa13c00b040 comp rx=0 tx=0).stop 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 10 monclient: shutdown 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 msgr2=0x7fa144133c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa14953e700 1 -- 172.21.15.5:0/1048578439 reap_dead start 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa12c016000 tx=0x7fa12c004770 comp rx=0 tx=0).stop 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 shutdown_connections 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 --2- 172.21.15.5:0/1048578439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa14409dfd0 0x7fa144133c60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 >> 172.21.15.5:0/1048578439 conn(0x7fa1440a45d0 msgr2=0x7fa14406d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 shutdown_connections 2024-09-16T11:02:13.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 -- 172.21.15.5:0/1048578439 wait complete. 2024-09-16T11:02:13.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:02:13.798+0000 7fa149d3f700 1 librados: shutdown 2024-09-16T11:02:13.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:02:13.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T11:03:43.840 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T11:03:43.840 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T11:03:43.840 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T11:03:43.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T11:03:43.844 INFO:tasks.ceph.mon.b:Started 2024-09-16T11:03:43.845 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:03:43.845 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:43.961 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T11:03:43.960+0000 7fdfbd32ad40 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 515 -> 0 (sync_force? mon rebuild?) 2024-09-16T11:03:43.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: get_monmap_and_config 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: build_initial_monmap 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: monmap: 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:43.961820+0000 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:43.961820+0000 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:43.963 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:43.964 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:43.964 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:43.964 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 Processor -- start 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 -- start start 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: init 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-c con 0x7f4e9809e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-b con 0x7f4e9809f200 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:43.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4e9813a460 0x7f4e9813a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-a con 0x7f4e9813a460 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9809c030 con 0x7f4e9813a460 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9809fb80 con 0x7f4e9809e8e0 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9809b970 con 0x7f4e9809f200 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9f099700 10 monclient(hunting): authenticate will time out at 1682.538302s 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:51782/0 (socket says 172.21.15.5:51782) 2024-09-16T11:03:43.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37954/0 (socket says 172.21.15.5:37954) 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 -- 172.21.15.5:0/1477168374 learned_addr learned my addr 172.21.15.5:0/1477168374 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 10 monclient(hunting): get_auth_request con 0x7f4e9809e8e0 auth_method 0 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): get_auth_request con 0x7f4e9809f200 auth_method 0 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 -- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 msgr2=0x7f4e9809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 -- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 msgr2=0x7f4e9809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:43.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e097700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e898700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.5:3300/0 conn(0x7f4e9813a460 0x7f4e9813a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e898700 10 monclient(hunting): get_auth_request con 0x7f4e9813a460 auth_method 0 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e898700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e898700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9e898700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient(hunting): handle_auth_done global_id 84209 payload 274 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 -- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 msgr2=0x7f4e9809ecc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 -- 172.21.15.5:0/1477168374 >> v2:172.21.15.5:3300/0 conn(0x7f4e9813a460 msgr2=0x7f4e9813a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:43.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.5:3300/0 conn(0x7f4e9813a460 0x7f4e9813a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient: _finish_hunting 0 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 monclient: found mon.noname-b 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 -- 172.21.15.5:0/1477168374 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9809bc50 con 0x7f4e9809f200 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.959+0000 7f4e9d896700 1 --2- 172.21.15.5:0/1477168374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e980a1180 tx=0x7f4e9400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1944b7acf7dafa94 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 1 -- 172.21.15.5:0/1477168374 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400e070 con 0x7f4e9809f200 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: dump: 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:43.977 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: _finish_auth 0 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: _check_auth_tickets 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 5 monclient: authenticate success, global_id 84209 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 1 -- 172.21.15.5:0/1477168374 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400bbd0 con 0x7f4e9809f200 2024-09-16T11:03:43.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 4 set_mon_vals no callback set 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 1 -- 172.21.15.5:0/1477168374 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94012430 con 0x7f4e9809f200 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d095700 10 monclient: dump: 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:43.979 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: shutdown 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/1477168374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 msgr2=0x7f4e98139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- 172.21.15.5:0/1477168374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4e980a1180 tx=0x7f4e9400b330 comp rx=0 tx=0).stop 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/1477168374 shutdown_connections 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.5:3300/0 conn(0x7f4e9813a460 0x7f4e9813a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- 172.21.15.5:0/1477168374 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809f200 0x7f4e98139f20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- 172.21.15.5:0/1477168374 >> v2:172.21.15.143:3300/0 conn(0x7f4e9809e8e0 0x7f4e9809ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/1477168374 >> 172.21.15.5:0/1477168374 conn(0x7f4e980a45d0 msgr2=0x7f4e980a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/1477168374 shutdown_connections 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/1477168374 wait complete. 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: build_initial_monmap 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: monmap: 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:43.981 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 librados: starting msgr at 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 librados: starting objecter 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 Processor -- start 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- start start 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 librados: setting wanted keys 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 librados: calling monclient init 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: init 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-c con 0x7f4e9809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 0x7f4e98139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-a con 0x7f4e9809f200 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): picked mon.noname-b con 0x7f4e9813a460 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9809b0d0 con 0x7f4e9809f200 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98139f00 con 0x7f4e9813a460 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9809d830 con 0x7f4e9809e8e0 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient(hunting): authenticate will time out at 1682.542447s 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 0x7f4e98139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37962/0 (socket says 172.21.15.5:37962) 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 -- 172.21.15.5:0/3471472016 learned_addr learned my addr 172.21.15.5:0/3471472016 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): get_auth_request con 0x7f4e9809e8e0 auth_method 0 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 0x7f4e98139480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57290/0 (socket says 172.21.15.5:57290) 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): get_auth_request con 0x7f4e9809f200 auth_method 0 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 10 monclient(hunting): get_auth_request con 0x7f4e9813a460 auth_method 0 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 1 -- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 msgr2=0x7f4e981399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 1 -- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 msgr2=0x7f4e981399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:43.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9d896700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e898700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient(hunting): handle_auth_done global_id 84212 payload 824 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 -- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 msgr2=0x7f4e981399c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 -- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 msgr2=0x7f4e98139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 0x7f4e98139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient: _finish_hunting 0 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 monclient: found mon.noname-c 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 -- 172.21.15.5:0/3471472016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94009a70 con 0x7f4e9809e8e0 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9e097700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800d800 tx=0x7f4e8800db10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 client.?.objecter ms_handle_connect 0x7f4e9809e8e0 2024-09-16T11:03:43.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88004020 con 0x7f4e9809e8e0 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: dump: 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:43.987 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: _finish_auth 0 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880041a0 con 0x7f4e9809e8e0 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 5 monclient: authenticate success, global_id 84212 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: _renew_subs 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e981248b0 con 0x7f4e9809e8e0 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88004350 con 0x7f4e9809e8e0 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:43.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e8effd700 10 monclient: dump: 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9c894700 4 set_mon_vals no callback set 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.963+0000 7f4e9f099700 10 client.84212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:43.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 10 monclient: _renew_subs 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e98126420 con 0x7f4e9809e8e0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 1 librados: init done 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e98126890 con 0x7f4e9809e8e0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f4e880036c0 con 0x7f4e9809e8e0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 1 --2- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 0x7f4e7c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(561..561 src has 395..561) v4 ==== 5732+0+0 (secure 0 0 0) 0x7f4e88047de0 con 0x7f4e9809e8e0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 10 client.84212.objecter ms_dispatch 0x7f4e98000fd0 osd_map(561..561 src has 395..561) v4 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 3 client.84212.objecter handle_osd_map got epochs [561,561] > 0 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 3 client.84212.objecter handle_osd_map decoding full epoch 561 2024-09-16T11:03:43.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 20 client.84212.objecter dump_active .. 0 homeless 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9d896700 1 --2- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 0x7f4e7c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9d896700 10 monclient: get_auth_request con 0x7f4e7c036f60 auth_method 0 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e8effd700 10 client.84212.objecter ms_handle_connect 0x7f4e7c036f60 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.967+0000 7f4e9d896700 1 --2- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 0x7f4e7c039420 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f4e94014040 tx=0x7f4e94011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.971+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4e98126890 con 0x7f4e9809e8e0 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.971+0000 7f4e8effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:43.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:43.971+0000 7f4e8effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4e98126890 con 0x7f4e9809e8e0 2024-09-16T11:03:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e8effd700 1 -- 172.21.15.5:0/3471472016 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f4e98126890 con 0x7f4e9809e8e0 2024-09-16T11:03:44.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e8effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:44.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e8effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:44.220 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 librados: watch_flush enter 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 librados: watch_flush exit 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 20 client.84212.objecter shutdown clearing up homeless session... 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 client.84212.objecter successfully canceled tick 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 msgr2=0x7f4e7c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 0x7f4e7c039420 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f4e94014040 tx=0x7f4e94011000 comp rx=0 tx=0).stop 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 10 monclient: shutdown 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 msgr2=0x7f4e981cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800d800 tx=0x7f4e8800db10 comp rx=0 tx=0).stop 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 shutdown_connections 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e9813a460 0x7f4e981399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4e7c036f60 0x7f4e7c039420 unknown :-1 s=CLOSED pgs=3860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9809f200 0x7f4e98139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.215+0000 7f4e9f099700 1 --2- 172.21.15.5:0/3471472016 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f4e9809e8e0 0x7f4e981cea70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.219+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 >> 172.21.15.5:0/3471472016 conn(0x7f4e980a45d0 msgr2=0x7f4e9809cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.219+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 shutdown_connections 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.219+0000 7f4e9f099700 1 -- 172.21.15.5:0/3471472016 wait complete. 2024-09-16T11:03:44.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:44.219+0000 7f4e9f099700 1 librados: shutdown 2024-09-16T11:03:47.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: get_monmap_and_config 2024-09-16T11:03:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: build_initial_monmap 2024-09-16T11:03:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: monmap: 2024-09-16T11:03:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:47.349278+0000 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:47.349278+0000 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:47.351 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 Processor -- start 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 -- start start 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: init 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T11:03:47.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f55780a2cf0 0x7f55780a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-a con 0x7f55780a2cf0 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55780a3e90 0x7f55780a3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-c con 0x7f55780a3e90 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5578135ab0 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780a4a80 con 0x7f5578135ab0 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780a3b50 con 0x7f55780a2cf0 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780a4310 con 0x7f55780a3e90 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient(hunting): authenticate will time out at 1685.926200s 2024-09-16T11:03:47.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f55780a2cf0 0x7f55780a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f55780a2cf0 0x7f55780a30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57298/0 (socket says 172.21.15.5:57298) 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 -- 172.21.15.5:0/3464715889 learned_addr learned my addr 172.21.15.5:0/3464715889 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): get_auth_request con 0x7f55780a2cf0 auth_method 0 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f55777fe700 1 --2- 172.21.15.5:0/3464715889 >> v2:172.21.15.143:3300/0 conn(0x7f55780a3e90 0x7f55780a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f55777fe700 10 monclient(hunting): get_auth_request con 0x7f55780a3e90 auth_method 0 2024-09-16T11:03:47.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f55777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient(hunting): handle_auth_done global_id 74379 payload 274 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 -- 172.21.15.5:0/3464715889 >> v2:172.21.15.143:3300/0 conn(0x7f55780a3e90 msgr2=0x7f55780a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 --2- 172.21.15.5:0/3464715889 >> v2:172.21.15.143:3300/0 conn(0x7f55780a3e90 0x7f55780a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 -- 172.21.15.5:0/3464715889 >> v1:172.21.15.143:6789/0 conn(0x7f5578135ab0 legacy=0x7f5578135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 monclient: found mon.noname-a 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 -- 172.21.15.5:0/3464715889 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780a06c0 con 0x7f55780a2cf0 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5577fff700 1 --2- 172.21.15.5:0/3464715889 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f55780a30d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f557809f0f0 tx=0x7f557000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d1455832fddc97 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:47.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 1 -- 172.21.15.5:0/3464715889 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557000e070 con 0x7f55780a2cf0 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: dump: 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:47.364 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: _finish_auth 0 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 5 monclient: authenticate success, global_id 74379 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 1 -- 172.21.15.5:0/3464715889 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570003a40 con 0x7f55780a2cf0 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f557d2eb700 4 set_mon_vals no callback set 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 1 -- 172.21.15.5:0/3464715889 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570003df0 con 0x7f55780a2cf0 2024-09-16T11:03:47.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.347+0000 7f5576ffd700 10 monclient: dump: 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:47.366 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: shutdown 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- 172.21.15.5:0/3464715889 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 msgr2=0x7f55780a30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- 172.21.15.5:0/3464715889 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f55780a30d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f557809f0f0 tx=0x7f557000b330 comp rx=0 tx=0).stop 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- 172.21.15.5:0/3464715889 shutdown_connections 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- 172.21.15.5:0/3464715889 >> v2:172.21.15.143:3300/0 conn(0x7f55780a3e90 0x7f55780a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- 172.21.15.5:0/3464715889 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f55780a30d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- 172.21.15.5:0/3464715889 >> 172.21.15.5:0/3464715889 conn(0x7f55780a89e0 msgr2=0x7f55780a4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- 172.21.15.5:0/3464715889 shutdown_connections 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- 172.21.15.5:0/3464715889 wait complete. 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: build_initial_monmap 2024-09-16T11:03:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: monmap: 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 librados: starting msgr at 2024-09-16T11:03:47.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 librados: starting objecter 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 Processor -- start 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- start start 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 librados: setting wanted keys 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 librados: calling monclient init 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: init 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-a con 0x7f55780a2cf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55780a3e90 0x7f5578133c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-c con 0x7f55780a3e90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): picked mon.noname-b con 0x7f5578135ab0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557811fa90 con 0x7f55780a2cf0 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557809f4e0 con 0x7f5578135ab0 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5578070f30 con 0x7f55780a3e90 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557d2eb700 10 monclient(hunting): authenticate will time out at 1685.930812s 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57308/0 (socket says 172.21.15.5:57308) 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 -- 172.21.15.5:0/618509966 learned_addr learned my addr 172.21.15.5:0/618509966 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:47.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): get_auth_request con 0x7f55780a2cf0 auth_method 0 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55780a3e90 0x7f5578133c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): get_auth_request con 0x7f55780a3e90 auth_method 0 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 10 monclient(hunting): get_auth_request con 0x7f5578135ab0 auth_method 0 2024-09-16T11:03:47.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f55777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 1 -- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 msgr2=0x7f55781341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 1 -- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 msgr2=0x7f55781341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f557caea700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient(hunting): handle_auth_done global_id 74382 payload 824 2024-09-16T11:03:47.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 -- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55780a3e90 msgr2=0x7f5578133c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55780a3e90 0x7f5578133c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 -- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 msgr2=0x7f55781341c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 monclient: found mon.noname-a 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.351+0000 7f5577fff700 1 -- 172.21.15.5:0/618509966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570009b30 con 0x7f55780a2cf0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5577fff700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5570000f80 tx=0x7f5570004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 client.?.objecter ms_handle_connect 0x7f55780a2cf0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570004440 con 0x7f55780a2cf0 2024-09-16T11:03:47.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: dump: 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:47.374 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: _finish_auth 0 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 5 monclient: authenticate success, global_id 74382 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _renew_subs 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55700045c0 con 0x7f55780a2cf0 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578120da0 con 0x7f55780a2cf0 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570004970 con 0x7f55780a2cf0 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:47.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 monclient: dump: 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:47.376 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 client.74382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _renew_subs 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578121130 con 0x7f55780a2cf0 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 1 librados: init done 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f55767fc700 4 set_mon_vals no callback set 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557811ff70 con 0x7f55780a2cf0 2024-09-16T11:03:47.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5570012430 con 0x7f55780a2cf0 2024-09-16T11:03:47.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 --2- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 0x7f5558039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:47.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(565..565 src has 395..565) v4 ==== 5839+0+0 (secure 0 0 0) 0x7f55700495d0 con 0x7f55780a2cf0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 10 client.74382.objecter ms_dispatch 0x7f5578000fd0 osd_map(565..565 src has 395..565) v4 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 3 client.74382.objecter handle_osd_map got epochs [565,565] > 0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 3 client.74382.objecter handle_osd_map decoding full epoch 565 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f5574ff9700 20 client.74382.objecter dump_active .. 0 homeless 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f55777fe700 1 --2- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 0x7f5558039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.355+0000 7f55777fe700 10 monclient: get_auth_request con 0x7f5558036f60 auth_method 0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.359+0000 7f5574ff9700 10 client.74382.objecter ms_handle_connect 0x7f5558036f60 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.359+0000 7f55777fe700 1 --2- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 0x7f5558039420 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7f55780a6840 tx=0x7f5560008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.367+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f557811ff70 con 0x7f55780a2cf0 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.367+0000 7f5574ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:47.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.367+0000 7f5574ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:47.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f557d2eb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:47.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f557d2eb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:47.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f557d2eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:47.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f557811ff70 con 0x7f55780a2cf0 2024-09-16T11:03:47.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f5574ff9700 1 -- 172.21.15.5:0/618509966 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f557811ff70 con 0x7f55780a2cf0 2024-09-16T11:03:47.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f5574ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:47.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.599+0000 7f5574ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:47.605 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 10 librados: watch_flush enter 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 10 librados: watch_flush exit 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 20 client.74382.objecter shutdown clearing up homeless session... 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 10 client.74382.objecter successfully canceled tick 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 msgr2=0x7f5558039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 0x7f5558039420 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7f55780a6840 tx=0x7f5560008040 comp rx=0 tx=0).stop 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 10 monclient: shutdown 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:47.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 msgr2=0x7f5578070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5570000f80 tx=0x7f5570004af0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 shutdown_connections 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5578135ab0 0x7f55781341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f55780a3e90 0x7f5578133c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5558036f60 0x7f5558039420 unknown :-1 s=CLOSED pgs=3861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 --2- 172.21.15.5:0/618509966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f55780a2cf0 0x7f5578070400 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 >> 172.21.15.5:0/618509966 conn(0x7f55780a89e0 msgr2=0x7f5578136300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 shutdown_connections 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 -- 172.21.15.5:0/618509966 wait complete. 2024-09-16T11:03:47.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:47.603+0000 7f557d2eb700 1 librados: shutdown 2024-09-16T11:03:50.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: get_monmap_and_config 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: build_initial_monmap 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: monmap: 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:50.739026+0000 2024-09-16T11:03:50.740 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:50.739026+0000 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:50.741 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 Processor -- start 2024-09-16T11:03:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 -- start start 2024-09-16T11:03:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: init 2024-09-16T11:03:50.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 0x7fd79809ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd79809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd79809d6a0 0x7fd79809f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd79809d6a0 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd79809dfd0 0x7fd7981336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd79809dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980a01d0 con 0x7fd79809d6a0 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd798070e40 con 0x7fd79809dfd0 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79809f740 con 0x7fd79809e8e0 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.735+0000 7fd797fff700 10 monclient(hunting): authenticate will time out at 1689.315818s 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd79809d6a0 0x7fd79809f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 0x7fd79809ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd79809d6a0 0x7fd79809f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57330/0 (socket says 172.21.15.5:57330) 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 0x7fd79809ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37982/0 (socket says 172.21.15.5:37982) 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 -- 172.21.15.5:0/3303683865 learned_addr learned my addr 172.21.15.5:0/3303683865 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): get_auth_request con 0x7fd79809d6a0 auth_method 0 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): get_auth_request con 0x7fd79809e8e0 auth_method 0 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7977fe700 1 --2- 172.21.15.5:0/3303683865 >> v2:172.21.15.143:3300/0 conn(0x7fd79809dfd0 0x7fd7981336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7977fe700 10 monclient(hunting): get_auth_request con 0x7fd79809dfd0 auth_method 0 2024-09-16T11:03:50.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient(hunting): handle_auth_done global_id 74385 payload 274 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 -- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 msgr2=0x7fd79809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 0x7fd79809ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 -- 172.21.15.5:0/3303683865 >> v2:172.21.15.143:3300/0 conn(0x7fd79809dfd0 msgr2=0x7fd7981336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3303683865 >> v2:172.21.15.143:3300/0 conn(0x7fd79809dfd0 0x7fd7981336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient: _finish_hunting 0 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 monclient: found mon.noname-a 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 -- 172.21.15.5:0/3303683865 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79809fef0 con 0x7fd79809d6a0 2024-09-16T11:03:50.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809d6a0 0x7fd79809f200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd798071dd0 tx=0x7fd788009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cea8c53c1c4a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 1 -- 172.21.15.5:0/3303683865 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788011830 con 0x7fd79809d6a0 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: dump: 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:50.750 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: _finish_auth 0 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 5 monclient: authenticate success, global_id 74385 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 1 -- 172.21.15.5:0/3303683865 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788011990 con 0x7fd79809d6a0 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 1 -- 172.21.15.5:0/3303683865 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788011d20 con 0x7fd79809d6a0 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd795ffb700 10 monclient: dump: 2024-09-16T11:03:50.751 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 4 set_mon_vals no callback set 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 10 monclient: shutdown 2024-09-16T11:03:50.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 -- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809d6a0 msgr2=0x7fd79809f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 --2- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809d6a0 0x7fd79809f200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd798071dd0 tx=0x7fd788009bf0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 -- 172.21.15.5:0/3303683865 shutdown_connections 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 --2- 172.21.15.5:0/3303683865 >> v2:172.21.15.143:3300/0 conn(0x7fd79809dfd0 0x7fd7981336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 --2- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809d6a0 0x7fd79809f200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 --2- 172.21.15.5:0/3303683865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809e8e0 0x7fd79809ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 -- 172.21.15.5:0/3303683865 >> 172.21.15.5:0/3303683865 conn(0x7fd7980a45d0 msgr2=0x7fd7980a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 -- 172.21.15.5:0/3303683865 shutdown_connections 2024-09-16T11:03:50.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.739+0000 7fd797fff700 1 -- 172.21.15.5:0/3303683865 wait complete. 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient: build_initial_monmap 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient: monmap: 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:50.759 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 librados: starting msgr at 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 librados: starting objecter 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 Processor -- start 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 -- start start 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 librados: setting wanted keys 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 librados: calling monclient init 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient: init 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:03:50.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd79809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd79809dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 0x7fd7981330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd79809e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981c8750 con 0x7fd79809e8e0 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7981c9140 con 0x7fd79809dfd0 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79811ed70 con 0x7fd79809d6a0 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:50.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd797fff700 10 monclient(hunting): authenticate will time out at 1689.322771s 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37998/0 (socket says 172.21.15.5:37998) 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 0x7fd7981330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 1 -- 172.21.15.5:0/3534760147 learned_addr learned my addr 172.21.15.5:0/3534760147 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): get_auth_request con 0x7fd79809d6a0 auth_method 0 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): get_auth_request con 0x7fd79809e8e0 auth_method 0 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 10 monclient(hunting): get_auth_request con 0x7fd79809dfd0 auth_method 0 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 1 -- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 msgr2=0x7fd798132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 1 -- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 msgr2=0x7fd798132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:50.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7967fc700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient(hunting): handle_auth_done global_id 74388 payload 824 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 -- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 msgr2=0x7fd7981c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 -- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 msgr2=0x7fd798132bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient: _finish_hunting 0 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 monclient: found mon.noname-a 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 -- 172.21.15.5:0/3534760147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788009570 con 0x7fd79809e8e0 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd796ffd700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:50.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd7977fe700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 0x7fd7981330f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd790007d60 tx=0x7fd79000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd77f7fe700 10 client.?.objecter ms_handle_connect 0x7fd79809e8e0 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.743+0000 7fd77f7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790018070 con 0x7fd79809e8e0 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: dump: 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:50.765 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79000ebe0 con 0x7fd79809e8e0 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 5 monclient: authenticate success, global_id 74388 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _renew_subs 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7980a3480 con 0x7fd79809e8e0 2024-09-16T11:03:50.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790003a40 con 0x7fd79809e8e0 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 monclient: dump: 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:50.767 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd7957fa700 4 set_mon_vals no callback set 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 client.74388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _renew_subs 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd798120290 con 0x7fd79809e8e0 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 1 librados: init done 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd790014420 con 0x7fd79809e8e0 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79806f760 con 0x7fd79809e8e0 2024-09-16T11:03:50.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 --2- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 0x7fd784039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(568..568 src has 395..568) v4 ==== 5402+0+0 (secure 0 0 0) 0x7fd790047f10 con 0x7fd79809e8e0 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 10 client.74388.objecter ms_dispatch 0x7fd798000fd0 osd_map(568..568 src has 395..568) v4 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 3 client.74388.objecter handle_osd_map got epochs [568,568] > 0 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 3 client.74388.objecter handle_osd_map decoding full epoch 568 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd77f7fe700 20 client.74388.objecter dump_active .. 0 homeless 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd796ffd700 1 --2- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 0x7fd784039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.747+0000 7fd796ffd700 10 monclient: get_auth_request con 0x7fd784036f60 auth_method 0 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.751+0000 7fd77f7fe700 10 client.74388.objecter ms_handle_connect 0x7fd784036f60 2024-09-16T11:03:50.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.751+0000 7fd796ffd700 1 --2- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 0x7fd784039420 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009fd0 tx=0x7fd78c009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:50.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.755+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd7900146d0 con 0x7fd79809e8e0 2024-09-16T11:03:50.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.755+0000 7fd77f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:50.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.755+0000 7fd77f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.991+0000 7fd797fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.991+0000 7fd797fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.991+0000 7fd797fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:50.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.991+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd7981336c0 con 0x7fd79809e8e0 2024-09-16T11:03:50.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd77f7fe700 1 -- 172.21.15.5:0/3534760147 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fd790049080 con 0x7fd79809e8e0 2024-09-16T11:03:50.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd77f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:50.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd77f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:50.998 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 10 librados: watch_flush enter 2024-09-16T11:03:51.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 10 librados: watch_flush exit 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 20 client.74388.objecter shutdown clearing up homeless session... 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 10 client.74388.objecter successfully canceled tick 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 msgr2=0x7fd784039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 0x7fd784039420 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7fd78c009fd0 tx=0x7fd78c009240 comp rx=0 tx=0).stop 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 10 monclient: shutdown 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 msgr2=0x7fd7981330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 0x7fd7981330f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd790007d60 tx=0x7fd79000e3c0 comp rx=0 tx=0).stop 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 shutdown_connections 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd784036f60 0x7fd784039420 unknown :-1 s=CLOSED pgs=3862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:51.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd79809e8e0 0x7fd7981330f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd79809dfd0 0x7fd798132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 --2- 172.21.15.5:0/3534760147 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd79809d6a0 0x7fd7981c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 >> 172.21.15.5:0/3534760147 conn(0x7fd7980a45d0 msgr2=0x7fd7980a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 shutdown_connections 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 -- 172.21.15.5:0/3534760147 wait complete. 2024-09-16T11:03:51.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:50.995+0000 7fd797fff700 1 librados: shutdown 2024-09-16T11:03:54.014 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.127+0000 7fad92ecc700 10 monclient: get_monmap_and_config 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.127+0000 7fad92ecc700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.127+0000 7fad92ecc700 10 monclient: monmap: 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:54.131778+0000 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:54.131778+0000 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:54.133 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 Processor -- start 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- start start 2024-09-16T11:03:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient: init 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-c con 0x7fad8c0a0970 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad8c09ca50 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-b con 0x7fad8c0a1290 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c09a440 con 0x7fad8c09ca50 2024-09-16T11:03:54.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c09f4f0 con 0x7fad8c0a0970 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c099d80 con 0x7fad8c0a1290 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient(hunting): authenticate will time out at 1692.709523s 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47594/0 (socket says 172.21.15.5:47594) 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 -- 172.21.15.5:0/3423593695 learned_addr learned my addr 172.21.15.5:0/3423593695 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): get_auth_request con 0x7fad8c0a1290 auth_method 0 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 1 --2- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 10 monclient(hunting): get_auth_request con 0x7fad8c0a0970 auth_method 0 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 1 -- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 msgr2=0x7fad8c0a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 1 -- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 msgr2=0x7fad8c0a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 1 --2- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad91eca700 1 --2- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient(hunting): handle_auth_done global_id 84224 payload 274 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 -- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 msgr2=0x7fad8c0a0d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 --2- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 -- 172.21.15.5:0/3423593695 >> v1:172.21.15.143:6789/0 conn(0x7fad8c09ca50 legacy=0x7fad8c132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 monclient: found mon.noname-b 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:54.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 -- 172.21.15.5:0/3423593695 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c09a1c0 con 0x7fad8c0a1290 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad926cb700 1 --2- 172.21.15.5:0/3423593695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fad8400d150 tx=0x7fad8400d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e5c9804a890ee6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 1 -- 172.21.15.5:0/3423593695 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84010070 con 0x7fad8c0a1290 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: dump: 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.143 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 5 monclient: authenticate success, global_id 84224 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 1 -- 172.21.15.5:0/3423593695 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8400ddc0 con 0x7fad8c0a1290 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 4 set_mon_vals no callback set 2024-09-16T11:03:54.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 1 -- 172.21.15.5:0/3423593695 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400b310 con 0x7fad8c0a1290 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad90ec8700 10 monclient: dump: 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.145 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 10 monclient: shutdown 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- 172.21.15.5:0/3423593695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 msgr2=0x7fad8c132e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 --2- 172.21.15.5:0/3423593695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fad8400d150 tx=0x7fad8400d460 comp rx=0 tx=0).stop 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- 172.21.15.5:0/3423593695 shutdown_connections 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 --2- 172.21.15.5:0/3423593695 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c0a1290 0x7fad8c132e30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 --2- 172.21.15.5:0/3423593695 >> v2:172.21.15.143:3300/0 conn(0x7fad8c0a0970 0x7fad8c0a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- 172.21.15.5:0/3423593695 >> 172.21.15.5:0/3423593695 conn(0x7fad8c0a2090 msgr2=0x7fad8c06a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- 172.21.15.5:0/3423593695 shutdown_connections 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.131+0000 7fad92ecc700 1 -- 172.21.15.5:0/3423593695 wait complete. 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient: monmap: 2024-09-16T11:03:54.146 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 librados: starting msgr at 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 librados: starting objecter 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 Processor -- start 2024-09-16T11:03:54.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 -- start start 2024-09-16T11:03:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 librados: setting wanted keys 2024-09-16T11:03:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 librados: calling monclient init 2024-09-16T11:03:54.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient: init 2024-09-16T11:03:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-c con 0x7fad8c09ca50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 0x7fad8c1317f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-a con 0x7fad8c0a0970 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): picked mon.noname-b con 0x7fad8c0a1290 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c1ced60 con 0x7fad8c0a0970 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c09c150 con 0x7fad8c0a1290 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c09e260 con 0x7fad8c09ca50 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad92ecc700 10 monclient(hunting): authenticate will time out at 1692.714615s 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 0x7fad8c1317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47606/0 (socket says 172.21.15.5:47606) 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 1 -- 172.21.15.5:0/1950120136 learned_addr learned my addr 172.21.15.5:0/1950120136 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): get_auth_request con 0x7fad8c09ca50 auth_method 0 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): get_auth_request con 0x7fad8c0a0970 auth_method 0 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 10 monclient(hunting): get_auth_request con 0x7fad8c0a1290 auth_method 0 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 1 -- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 msgr2=0x7fad8c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 1 -- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 msgr2=0x7fad8c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad926cb700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient(hunting): handle_auth_done global_id 74391 payload 824 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 -- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 msgr2=0x7fad8c1ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 -- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 msgr2=0x7fad8c131d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 monclient: found mon.noname-a 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 -- 172.21.15.5:0/1950120136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8400bd40 con 0x7fad8c0a0970 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad91eca700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad916c9700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 0x7fad8c1317f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c3c0 tx=0x7fad7c00c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad827fc700 10 client.?.objecter ms_handle_connect 0x7fad8c0a0970 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.135+0000 7fad827fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0052d0 con 0x7fad8c0a0970 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: dump: 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.156 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 5 monclient: authenticate success, global_id 74391 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c005450 con 0x7fad8c0a0970 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c006ac0 con 0x7fad8c0a0970 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _renew_subs 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c1219f0 con 0x7fad8c0a0970 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad83fff700 4 set_mon_vals no callback set 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 client.74391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 monclient: dump: 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _renew_subs 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c0a46a0 con 0x7fad8c0a0970 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 1 librados: init done 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fad7c0036a0 con 0x7fad8c0a0970 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c0a0d50 con 0x7fad8c0a0970 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 --2- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 0x7fad70039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(572..572 src has 395..572) v4 ==== 5755+0+0 (secure 0 0 0) 0x7fad7c047060 con 0x7fad8c0a0970 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 10 client.74391.objecter ms_dispatch 0x7fad8c000fd0 osd_map(572..572 src has 395..572) v4 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 3 client.74391.objecter handle_osd_map got epochs [572,572] > 0 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 3 client.74391.objecter handle_osd_map decoding full epoch 572 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad827fc700 20 client.74391.objecter dump_active .. 0 homeless 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad91eca700 1 --2- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 0x7fad70039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.139+0000 7fad91eca700 10 monclient: get_auth_request con 0x7fad70036f60 auth_method 0 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.143+0000 7fad827fc700 10 client.74391.objecter ms_handle_connect 0x7fad70036f60 2024-09-16T11:03:54.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.143+0000 7fad91eca700 1 --2- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 0x7fad70039420 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7fad88006f30 tx=0x7fad88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.147+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad7c019900 con 0x7fad8c0a0970 2024-09-16T11:03:54.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.147+0000 7fad827fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.147+0000 7fad827fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:54.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad92ecc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:54.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad92ecc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad92ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad8c121250 con 0x7fad8c0a0970 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad827fc700 1 -- 172.21.15.5:0/1950120136 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fad7c019310 con 0x7fad8c0a0970 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad827fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.391+0000 7fad827fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:54.396 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 10 librados: watch_flush enter 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 10 librados: watch_flush exit 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 20 client.74391.objecter shutdown clearing up homeless session... 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 10 client.74391.objecter successfully canceled tick 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 msgr2=0x7fad70039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 0x7fad70039420 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7fad88006f30 tx=0x7fad88008040 comp rx=0 tx=0).stop 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 10 monclient: shutdown 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 msgr2=0x7fad8c1317f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 0x7fad8c1317f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c3c0 tx=0x7fad7c00c6d0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 shutdown_connections 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad8c0a1290 0x7fad8c131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> v2:172.21.15.143:6812/1098205034 conn(0x7fad70036f60 0x7fad70039420 unknown :-1 s=CLOSED pgs=3863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad8c0a0970 0x7fad8c1317f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 --2- 172.21.15.5:0/1950120136 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fad8c09ca50 0x7fad8c1ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 >> 172.21.15.5:0/1950120136 conn(0x7fad8c0a2090 msgr2=0x7fad8c09e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 shutdown_connections 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 -- 172.21.15.5:0/1950120136 wait complete. 2024-09-16T11:03:54.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.395+0000 7fad92ecc700 1 librados: shutdown 2024-09-16T11:03:54.410 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:03:54.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:03:54.410 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 10 monclient: get_monmap_and_config 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 10 monclient: monmap: 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:54.523775+0000 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:54.523775+0000 2024-09-16T11:03:54.525 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:54.526 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 1 Processor -- start 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 1 -- start start 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.519+0000 7fc12974b700 10 monclient: init 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409e8e0 0x7fc12409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-b con 0x7fc12409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-c con 0x7fc12409d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc12409dfd0 0x7fc1241336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-a con 0x7fc12409dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1240a01d0 con 0x7fc12409dfd0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc124070e40 con 0x7fc12409d6a0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12409f740 con 0x7fc12409e8e0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient(hunting): authenticate will time out at 1693.100136s 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc12409dfd0 0x7fc1241336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409e8e0 0x7fc12409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc12409dfd0 0x7fc1241336c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57848/0 (socket says 172.21.15.5:57848) 2024-09-16T11:03:54.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 -- 172.21.15.5:0/2840712689 learned_addr learned my addr 172.21.15.5:0/2840712689 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): get_auth_request con 0x7fc12409dfd0 auth_method 0 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): get_auth_request con 0x7fc12409e8e0 auth_method 0 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 10 monclient(hunting): get_auth_request con 0x7fc12409d6a0 auth_method 0 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient(hunting): handle_auth_done global_id 74394 payload 274 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 -- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409e8e0 msgr2=0x7fc12409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409e8e0 0x7fc12409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 -- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 msgr2=0x7fc12409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:54.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 1 -- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 msgr2=0x7fc12409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 1 -- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 msgr2=0x7fc12409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 monclient: found mon.noname-a 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 -- 172.21.15.5:0/2840712689 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12409fef0 con 0x7fc12409dfd0 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409dfd0 0x7fc1241336c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1240a13c0 tx=0x7fc11c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88fb2ec421b364f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 1 -- 172.21.15.5:0/2840712689 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00e070 con 0x7fc12409dfd0 2024-09-16T11:03:54.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: dump: 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 5 monclient: authenticate success, global_id 74394 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 1 -- 172.21.15.5:0/2840712689 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c00b910 con 0x7fc12409dfd0 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 4 set_mon_vals no callback set 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 1 -- 172.21.15.5:0/2840712689 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00bcc0 con 0x7fc12409dfd0 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc1237fe700 10 monclient: dump: 2024-09-16T11:03:54.543 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: shutdown 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409dfd0 msgr2=0x7fc1241336c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409dfd0 0x7fc1241336c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1240a13c0 tx=0x7fc11c00b330 comp rx=0 tx=0).stop 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- 172.21.15.5:0/2840712689 shutdown_connections 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409dfd0 0x7fc1241336c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- 172.21.15.5:0/2840712689 >> v2:172.21.15.143:3300/0 conn(0x7fc12409d6a0 0x7fc12409f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 --2- 172.21.15.5:0/2840712689 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409e8e0 0x7fc12409ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- 172.21.15.5:0/2840712689 >> 172.21.15.5:0/2840712689 conn(0x7fc1240a45d0 msgr2=0x7fc1240a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- 172.21.15.5:0/2840712689 shutdown_connections 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 -- 172.21.15.5:0/2840712689 wait complete. 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 10 monclient: monmap: 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:54.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 librados: starting msgr at 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.523+0000 7fc12974b700 1 librados: starting objecter 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 Processor -- start 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- start start 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 librados: setting wanted keys 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 librados: calling monclient init 2024-09-16T11:03:54.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: init 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 0x7fc1241ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-a con 0x7fc12409d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409dfd0 0x7fc124131c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-c con 0x7fc12409dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 0x7fc1241c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): picked mon.noname-b con 0x7fc12409e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12409c890 con 0x7fc12409d6a0 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124121060 con 0x7fc12409e8e0 2024-09-16T11:03:54.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc124121340 con 0x7fc12409dfd0 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): authenticate will time out at 1693.105179s 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 0x7fc1241ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 0x7fc1241ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57850/0 (socket says 172.21.15.5:57850) 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 -- 172.21.15.5:0/2890785653 learned_addr learned my addr 172.21.15.5:0/2890785653 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): get_auth_request con 0x7fc12409d6a0 auth_method 0 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409dfd0 0x7fc124131c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): get_auth_request con 0x7fc12409dfd0 auth_method 0 2024-09-16T11:03:54.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 0x7fc1241c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 10 monclient(hunting): get_auth_request con 0x7fc12409e8e0 auth_method 0 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient(hunting): handle_auth_done global_id 74397 payload 824 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 msgr2=0x7fc1241c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 msgr2=0x7fc1241c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 0x7fc1241c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409dfd0 msgr2=0x7fc124131c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc12409dfd0 0x7fc124131c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 0x7fc1241c6e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 msgr2=0x7fc1241c6e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc12409e8e0 0x7fc1241c6e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 monclient: found mon.noname-a 2024-09-16T11:03:54.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c009b30 con 0x7fc12409d6a0 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 client.?.objecter ms_handle_connect 0x7fc12409d6a0 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc123fff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 0x7fc1241ca310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00e9d0 tx=0x7fc10c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c009720 con 0x7fc12409d6a0 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: dump: 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.551 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 5 monclient: authenticate success, global_id 74397 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _renew_subs 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124121620 con 0x7fc12409d6a0 2024-09-16T11:03:54.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c00c850 con 0x7fc12409d6a0 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00cbc0 con 0x7fc12409d6a0 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc1217fa700 10 monclient: dump: 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.553 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc122ffd700 4 set_mon_vals no callback set 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 client.74397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _renew_subs 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12411fa30 con 0x7fc12409d6a0 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 librados: init done 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 0x7fc1241c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124067380 con 0x7fc124132180 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 0x7fc1241c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient: get_auth_request con 0x7fc124132180 auth_method 0 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.527+0000 7fc11bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 10 monclient(hunting): handle_auth_done global_id 74400 payload 824 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc12409db00 con 0x7fc124132180 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc10c00cd20 con 0x7fc12409d6a0 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 --2- 172.21.15.5:0/2890785653 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc104036f60 0x7fc104039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(572..572 src has 395..572) v4 ==== 5755+0+0 (secure 0 0 0) 0x7fc10c048010 con 0x7fc12409d6a0 2024-09-16T11:03:54.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 client.74397.objecter ms_dispatch 0x7fc124000fd0 osd_map(572..572 src has 395..572) v4 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 3 client.74397.objecter handle_osd_map got epochs [572,572] > 0 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 3 client.74397.objecter handle_osd_map decoding full epoch 572 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 0x7fc1241c7350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc124120c70 tx=0x7fc114011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 20 client.74397.objecter dump_active .. 0 homeless 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 client.74397.objecter ms_handle_connect 0x7fc124132180 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 client.74397.objecter resend_mon_ops 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11400c070 con 0x7fc124132180 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 monclient: dump: 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.556 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2890785653 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc104036f60 0x7fc104039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc128f4a700 10 monclient: get_auth_request con 0x7fc104036f60 auth_method 0 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fc114004230 con 0x7fc124132180 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 msgr2=0x7fc1241c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 0x7fc1241c7350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc124120c70 tx=0x7fc114011b90 comp rx=0 tx=0).stop 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 msgr2=0x7fc1241c7350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc124132180 0x7fc1241c7350 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc1217fa700 10 client.74397.objecter ms_handle_connect 0x7fc104036f60 2024-09-16T11:03:54.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.531+0000 7fc128f4a700 1 --2- 172.21.15.5:0/2890785653 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc104036f60 0x7fc104039420 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fc12406b9c0 tx=0x7fc11c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:54.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 0x7fc1241c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124132640 con 0x7fc12411fe90 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 0x7fc1241c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient: get_auth_request con 0x7fc12411fe90 auth_method 0 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 10 monclient(hunting): handle_auth_done global_id 74403 payload 824 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 1 -- 172.21.15.5:0/2890785653 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc1241336c0 con 0x7fc12411fe90 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc11bfff700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 0x7fc1241c7890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc124120c70 tx=0x7fc114004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 client.74397.objecter ms_handle_connect 0x7fc12411fe90 2024-09-16T11:03:54.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 client.74397.objecter resend_mon_ops 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc114028400 con 0x7fc12411fe90 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 monclient: dump: 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.562 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7fc114013400 con 0x7fc12411fe90 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 msgr2=0x7fc1241c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 0x7fc1241c7890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc124120c70 tx=0x7fc114004770 comp rx=0 tx=0).stop 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 msgr2=0x7fc1241c7890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc1217fa700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12411fe90 0x7fc1241c7890 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:54.564 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.565 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.566 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:54.567 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.568 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:54.569 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.570 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.571 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.572 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.573 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.574 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.575 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T11:03:54.576 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.577 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.577 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.577 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:54.577 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 librados: watch_flush enter 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 librados: watch_flush exit 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 20 client.74397.objecter shutdown clearing up homeless session... 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 client.74397.objecter successfully canceled tick 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc104036f60 msgr2=0x7fc104039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 --2- 172.21.15.5:0/2890785653 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc104036f60 0x7fc104039420 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7fc12406b9c0 tx=0x7fc11c009000 comp rx=0 tx=0).stop 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 10 monclient: shutdown 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 msgr2=0x7fc1241ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 --2- 172.21.15.5:0/2890785653 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc12409d6a0 0x7fc1241ca310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00e9d0 tx=0x7fc10c00ece0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc128f4a700 1 -- 172.21.15.5:0/2890785653 reap_dead start 2024-09-16T11:03:54.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc128f4a700 1 -- 172.21.15.5:0/2890785653 reap_dead start 2024-09-16T11:03:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 shutdown_connections 2024-09-16T11:03:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 >> 172.21.15.5:0/2890785653 conn(0x7fc1240a45d0 msgr2=0x7fc1240a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 shutdown_connections 2024-09-16T11:03:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 -- 172.21.15.5:0/2890785653 wait complete. 2024-09-16T11:03:54.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.539+0000 7fc12974b700 1 librados: shutdown 2024-09-16T11:03:54.580 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:54.580 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:03:54.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: get_monmap_and_config 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: monmap: 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:54.697603+0000 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:54.697603+0000 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.699 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:54.700 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 Processor -- start 2024-09-16T11:03:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 -- start start 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: init 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe2c409e8e0 0x7fe2c409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-a con 0x7fe2c409e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2c409d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2c409dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c4070e40 con 0x7fe2c409d6a0 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c41344e0 con 0x7fe2c409dfd0 2024-09-16T11:03:54.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c40a01d0 con 0x7fe2c409e8e0 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2cabcd700 10 monclient(hunting): authenticate will time out at 1693.274791s 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe2c409e8e0 0x7fe2c409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe2c409e8e0 0x7fe2c409ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57870/0 (socket says 172.21.15.5:57870) 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/3867974729 learned_addr learned my addr 172.21.15.5:0/3867974729 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request con 0x7fe2c409e8e0 auth_method 0 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c93ca700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe2c409d6a0 0x7fe2c41336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49492/0 (socket says 172.21.15.5:49492) 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_done global_id 74406 payload 274 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/3867974729 >> v1:172.21.15.143:6789/0 conn(0x7fe2c409dfd0 legacy=0x7fe2c4133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/3867974729 >> v1:172.21.15.5:6789/0 conn(0x7fe2c409d6a0 legacy=0x7fe2c41336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 monclient: found mon.noname-a 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/3867974729 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c409fef0 con 0x7fe2c409e8e0 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.695+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/3867974729 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409e8e0 0x7fe2c409ecc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009e80 tx=0x7fe2c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b215729c762eaa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 1 -- 172.21.15.5:0/3867974729 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c000e070 con 0x7fe2c409e8e0 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: dump: 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.707 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 5 monclient: authenticate success, global_id 74406 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 1 -- 172.21.15.5:0/3867974729 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c0003a40 con 0x7fe2c409e8e0 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 1 -- 172.21.15.5:0/3867974729 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0003df0 con 0x7fe2c409e8e0 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 4 set_mon_vals no callback set 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2c8bc9700 10 monclient: dump: 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.709 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 10 monclient: shutdown 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- 172.21.15.5:0/3867974729 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409e8e0 msgr2=0x7fe2c409ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/3867974729 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409e8e0 0x7fe2c409ecc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009e80 tx=0x7fe2c000b330 comp rx=0 tx=0).stop 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- 172.21.15.5:0/3867974729 shutdown_connections 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/3867974729 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409e8e0 0x7fe2c409ecc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- 172.21.15.5:0/3867974729 >> 172.21.15.5:0/3867974729 conn(0x7fe2c40a45d0 msgr2=0x7fe2c40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- 172.21.15.5:0/3867974729 shutdown_connections 2024-09-16T11:03:54.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- 172.21.15.5:0/3867974729 wait complete. 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 10 monclient: monmap: 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:54.717 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 librados: starting msgr at 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 librados: starting objecter 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 Processor -- start 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 -- start start 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 librados: setting wanted keys 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 1 librados: calling monclient init 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.699+0000 7fe2cabcd700 10 monclient: init 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 0x7fe2c41ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-b con 0x7fe2c409d6a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409dfd0 0x7fe2c4132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-a con 0x7fe2c409dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c409e8e0 0x7fe2c406e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): picked mon.noname-c con 0x7fe2c409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c409b910 con 0x7fe2c409dfd0 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c409b3e0 con 0x7fe2c409d6a0 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c409b0d0 con 0x7fe2c409e8e0 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): authenticate will time out at 1693.280363s 2024-09-16T11:03:54.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409dfd0 0x7fe2c4132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c409e8e0 0x7fe2c406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c409e8e0 0x7fe2c406e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47626/0 (socket says 172.21.15.5:47626) 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 1 -- 172.21.15.5:0/78668318 learned_addr learned my addr 172.21.15.5:0/78668318 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): get_auth_request con 0x7fe2c409dfd0 auth_method 0 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 0x7fe2c41ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): get_auth_request con 0x7fe2c409e8e0 auth_method 0 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request con 0x7fe2c409d6a0 auth_method 0 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 msgr2=0x7fe2c41ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:03:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 msgr2=0x7fe2c41ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 0x7fe2c41ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 0x7fe2c41ca440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient(hunting): handle_auth_done global_id 74412 payload 824 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c409e8e0 msgr2=0x7fe2c406e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c409e8e0 0x7fe2c406e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 msgr2=0x7fe2c41ca440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c409d6a0 0x7fe2c41ca440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 monclient: found mon.noname-a 2024-09-16T11:03:54.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c0009b30 con 0x7fe2c409dfd0 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c93ca700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409dfd0 0x7fe2c4132ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c500 tx=0x7fe2b400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 client.?.objecter ms_handle_connect 0x7fe2c409dfd0 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4005cd0 con 0x7fe2c409dfd0 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: dump: 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 5 monclient: authenticate success, global_id 74412 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4005e30 con 0x7fe2c409dfd0 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _renew_subs 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b40072d0 con 0x7fe2c409dfd0 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2bbfff700 4 set_mon_vals no callback set 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c41208b0 con 0x7fe2c409dfd0 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 monclient: dump: 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 client.74412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _renew_subs 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c4133860 con 0x7fe2c409dfd0 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 librados: init done 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe2b4007430 con 0x7fe2c409dfd0 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 0x7fe2c406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c4067380 con 0x7fe2c4133430 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 --2- 172.21.15.5:0/78668318 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe2a8036f90 0x7fe2a8039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 0x7fe2c406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(572..572 src has 395..572) v4 ==== 5755+0+0 (secure 0 0 0) 0x7fe2b4047260 con 0x7fe2c409dfd0 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient: get_auth_request con 0x7fe2c4133430 auth_method 0 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 10 client.74412.objecter ms_dispatch 0x7fe2c4000fd0 osd_map(572..572 src has 395..572) v4 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 3 client.74412.objecter handle_osd_map got epochs [572,572] > 0 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 3 client.74412.objecter handle_osd_map decoding full epoch 572 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ba7fc700 20 client.74412.objecter dump_active .. 0 homeless 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 1 --2- 172.21.15.5:0/78668318 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe2a8036f90 0x7fe2a8039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.703+0000 7fe2ca3cc700 10 monclient: get_auth_request con 0x7fe2a8036f90 auth_method 0 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_done global_id 84239 payload 824 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2c409f200 con 0x7fe2c4133430 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 client.74412.objecter ms_handle_connect 0x7fe2c4133430 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 client.74412.objecter resend_mon_ops 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 0x7fe2c406ebc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009e80 tx=0x7fe2c0004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 client.74412.objecter ms_handle_connect 0x7fe2a8036f90 2024-09-16T11:03:54.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ca3cc700 1 --2- 172.21.15.5:0/78668318 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe2a8036f90 0x7fe2a8039450 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc001910 tx=0x7fe2bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0004340 con 0x7fe2c4133430 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 monclient: dump: 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.729 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe2c00044a0 con 0x7fe2c4133430 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 msgr2=0x7fe2c406ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 0x7fe2c406ebc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009e80 tx=0x7fe2c0004af0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 msgr2=0x7fe2c406ebc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.707+0000 7fe2ba7fc700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c4133430 0x7fe2c406ebc0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009e80 tx=0x7fe2c0004af0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2cabcd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 0x7fe2c41b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2cabcd700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c406f510 con 0x7fe2c406f100 2024-09-16T11:03:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 0x7fe2c41b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient: get_auth_request con 0x7fe2c406f100 auth_method 0 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 10 monclient(hunting): handle_auth_done global_id 84242 payload 824 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 1 -- 172.21.15.5:0/78668318 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe2c4120010 con 0x7fe2c406f100 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2ba7fc700 10 client.74412.objecter ms_handle_connect 0x7fe2c406f100 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2ba7fc700 10 client.74412.objecter resend_mon_ops 2024-09-16T11:03:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.715+0000 7fe2c9bcb700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 0x7fe2c41b9870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009fd0 tx=0x7fe2c0011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0012430 con 0x7fe2c406f100 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 10 monclient: dump: 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7fe2c0012590 con 0x7fe2c406f100 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 msgr2=0x7fe2c41b9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 0x7fe2c41b9870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009fd0 tx=0x7fe2c0011000 comp rx=0 tx=0).stop 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 msgr2=0x7fe2c41b9870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ba7fc700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe2c406f100 0x7fe2c41b9870 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009fd0 tx=0x7fe2c0011000 comp rx=0 tx=0).stop 2024-09-16T11:03:54.735 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.736 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:54.737 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.738 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:54.739 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.740 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.741 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.742 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.743 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.744 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.745 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.747 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.748 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:54.749 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 10 librados: watch_flush enter 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 10 librados: watch_flush exit 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 20 client.74412.objecter shutdown clearing up homeless session... 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 10 client.74412.objecter successfully canceled tick 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe2a8036f90 msgr2=0x7fe2a8039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/78668318 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe2a8036f90 0x7fe2a8039450 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc001910 tx=0x7fe2bc009040 comp rx=0 tx=0).stop 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 10 monclient: shutdown 2024-09-16T11:03:54.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409dfd0 msgr2=0x7fe2c4132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 --2- 172.21.15.5:0/78668318 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2c409dfd0 0x7fe2c4132ef0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c500 tx=0x7fe2b400c810 comp rx=0 tx=0).stop 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ca3cc700 1 -- 172.21.15.5:0/78668318 reap_dead start 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2ca3cc700 1 -- 172.21.15.5:0/78668318 reap_dead start 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 shutdown_connections 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 >> 172.21.15.5:0/78668318 conn(0x7fe2c40a45d0 msgr2=0x7fe2c4133c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 shutdown_connections 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 -- 172.21.15.5:0/78668318 wait complete. 2024-09-16T11:03:54.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.719+0000 7fe2cabcd700 1 librados: shutdown 2024-09-16T11:03:54.752 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:54.753 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:03:54.753 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:03:54.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.867+0000 7f8013599700 10 monclient: get_monmap_and_config 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.867+0000 7f8013599700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.867+0000 7f8013599700 10 monclient: monmap: 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:54.872643+0000 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:54.872643+0000 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.874 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:54.875 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 Processor -- start 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 -- start start 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient: init 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): picked mon.noname-b con 0x7f800c09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f800c09d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f800c09dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c070e40 con 0x7f800c09d6a0 2024-09-16T11:03:54.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c1344e0 con 0x7f800c09dfd0 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0a01d0 con 0x7f800c09e8e0 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient(hunting): authenticate will time out at 1693.449586s 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011d96700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f800c09d6a0 0x7f800c1336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49498/0 (socket says 172.21.15.5:49498) 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011d96700 1 -- 172.21.15.5:0/3117977456 learned_addr learned my addr 172.21.15.5:0/3117977456 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): get_auth_request con 0x7f800c09e8e0 auth_method 0 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 1 -- 172.21.15.5:0/3117977456 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419419206 0 0) 0x7f800c070e40 con 0x7f800c09d6a0 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient(hunting): _negotiate 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient(hunting): my global_id is 74415 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 1 -- 172.21.15.5:0/3117977456 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80035d0 con 0x7f800c09d6a0 2024-09-16T11:03:54.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient(hunting): handle_auth_done global_id 84245 payload 274 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 -- 172.21.15.5:0/3117977456 >> v1:172.21.15.143:6789/0 conn(0x7f800c09dfd0 legacy=0x7f800c133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 -- 172.21.15.5:0/3117977456 >> v1:172.21.15.5:6789/0 conn(0x7f800c09d6a0 legacy=0x7f800c1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 monclient: found mon.noname-b 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 -- 172.21.15.5:0/3117977456 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c09fef0 con 0x7f800c09e8e0 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8012597700 1 --2- 172.21.15.5:0/3117977456 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c09ecc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8008009880 tx=0x7f8008009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3afb164d892bbb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 1 -- 172.21.15.5:0/3117977456 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008011830 con 0x7f800c09e8e0 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: dump: 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 5 monclient: authenticate success, global_id 84245 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 1 -- 172.21.15.5:0/3117977456 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8008011990 con 0x7f800c09e8e0 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8013599700 4 set_mon_vals no callback set 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 1 -- 172.21.15.5:0/3117977456 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008011d20 con 0x7f800c09e8e0 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:54.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.871+0000 7f8011595700 10 monclient: dump: 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 10 monclient: shutdown 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- 172.21.15.5:0/3117977456 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 msgr2=0x7f800c09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 --2- 172.21.15.5:0/3117977456 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c09ecc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8008009880 tx=0x7f8008009b90 comp rx=0 tx=0).stop 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- 172.21.15.5:0/3117977456 shutdown_connections 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 --2- 172.21.15.5:0/3117977456 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c09ecc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- 172.21.15.5:0/3117977456 >> 172.21.15.5:0/3117977456 conn(0x7f800c0a45d0 msgr2=0x7f800c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- 172.21.15.5:0/3117977456 shutdown_connections 2024-09-16T11:03:54.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- 172.21.15.5:0/3117977456 wait complete. 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 10 monclient: build_initial_monmap 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 10 monclient: monmap: 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:54.893 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 librados: starting msgr at 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 librados: starting objecter 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 Processor -- start 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 -- start start 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 librados: setting wanted keys 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 1 librados: calling monclient init 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.875+0000 7f8013599700 10 monclient: init 2024-09-16T11:03:54.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 0x7f800c1c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): picked mon.noname-a con 0x7f800c09d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c09dfd0 0x7f800c132bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): picked mon.noname-b con 0x7f800c09dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): picked mon.noname-c con 0x7f800c09e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c1c91f0 con 0x7f800c09d6a0 2024-09-16T11:03:54.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c11e220 con 0x7f800c09dfd0 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c133640 con 0x7f800c09e8e0 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): authenticate will time out at 1693.456270s 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 0x7f800c1c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c09dfd0 0x7f800c132bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 0x7f800c1c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57892/0 (socket says 172.21.15.5:57892) 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 1 -- 172.21.15.5:0/1940574044 learned_addr learned my addr 172.21.15.5:0/1940574044 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:54.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): get_auth_request con 0x7f800c09d6a0 auth_method 0 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c133100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47672/0 (socket says 172.21.15.5:47672) 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): get_auth_request con 0x7f800c09e8e0 auth_method 0 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): get_auth_request con 0x7f800c09dfd0 auth_method 0 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient(hunting): handle_auth_done global_id 84248 payload 824 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c09dfd0 msgr2=0x7f800c132bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c09dfd0 0x7f800c132bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 msgr2=0x7f800c1c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 0x7f800c1c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient: _finish_hunting 0 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 monclient: found mon.noname-c 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080094b0 con 0x7f800c09e8e0 2024-09-16T11:03:54.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012597700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f800c09d6a0 0x7f800c1c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8012d98700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c133100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f800400cb90 tx=0x7f800400cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 client.?.objecter ms_handle_connect 0x7f800c09e8e0 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004003c20 con 0x7f800c09e8e0 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: dump: 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: _finish_auth 0 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 5 monclient: authenticate success, global_id 84248 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8004003d80 con 0x7f800c09e8e0 2024-09-16T11:03:54.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _renew_subs 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c06c6b0 con 0x7f800c09e8e0 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400aa40 con 0x7f800c09e8e0 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8010d94700 4 set_mon_vals no callback set 2024-09-16T11:03:54.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 client.84248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 monclient: dump: 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.903 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _renew_subs 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c120580 con 0x7f800c09e8e0 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 librados: init done 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f800400aba0 con 0x7f800c09e8e0 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 --2- 172.21.15.5:0/1940574044 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7ff0036f60 0x7f7ff0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 0x7f800c11ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c06cc30 con 0x7f800c11c670 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(572..572 src has 395..572) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f8004047eb0 con 0x7f800c09e8e0 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 10 client.84248.objecter ms_dispatch 0x7f800c000fd0 osd_map(572..572 src has 395..572) v4 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 3 client.84248.objecter handle_osd_map got epochs [572,572] > 0 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.879+0000 7f8002ffd700 3 client.84248.objecter handle_osd_map decoding full epoch 572 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 20 client.84248.objecter dump_active .. 0 homeless 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 0x7f800c11ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8012597700 1 --2- 172.21.15.5:0/1940574044 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7ff0036f60 0x7f7ff0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8012597700 10 monclient: get_auth_request con 0x7f7ff0036f60 auth_method 0 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient: get_auth_request con 0x7f800c11c670 auth_method 0 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 10 monclient(hunting): handle_auth_done global_id 94102 payload 824 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f800c04ecd0 con 0x7f800c11c670 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 client.84248.objecter ms_handle_connect 0x7f7ff0036f60 2024-09-16T11:03:54.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8012597700 1 --2- 172.21.15.5:0/1940574044 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7ff0036f60 0x7f7ff0039420 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f80080097d0 tx=0x7f8008010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 client.84248.objecter ms_handle_connect 0x7f800c11c670 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 client.84248.objecter resend_mon_ops 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8011d96700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 0x7f800c11ca50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f800c133b80 tx=0x7f7ffc001950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc012560 con 0x7f800c11c670 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 monclient: dump: 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f7ffc0126c0 con 0x7f800c11c670 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 msgr2=0x7f800c11ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 0x7f800c11ca50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f800c133b80 tx=0x7f7ffc001950 comp rx=0 tx=0).stop 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 msgr2=0x7f800c11ca50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.883+0000 7f8002ffd700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11c670 0x7f800c11ca50 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f800c133b80 tx=0x7f7ffc001950 comp rx=0 tx=0).stop 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.891+0000 7f8013599700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.891+0000 7f8013599700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 0x7f800c06dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.891+0000 7f8013599700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.891+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c067380 con 0x7f800c11cf90 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 0x7f800c06dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient: get_auth_request con 0x7f800c11cf90 auth_method 0 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:54.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 10 monclient(hunting): handle_auth_done global_id 94105 payload 824 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 1 -- 172.21.15.5:0/1940574044 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f800c09db80 con 0x7f800c11cf90 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8011d96700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 0x7f800c06dbf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f800c11f480 tx=0x7f7ffc001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 client.84248.objecter ms_handle_connect 0x7f800c11cf90 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 client.84248.objecter resend_mon_ops 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc022070 con 0x7f800c11cf90 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 monclient: dump: 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:54.911 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f7ffc01d240 con 0x7f800c11cf90 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 msgr2=0x7f800c06dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 0x7f800c06dbf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f800c11f480 tx=0x7f7ffc001150 comp rx=0 tx=0).stop 2024-09-16T11:03:54.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 msgr2=0x7f800c06dbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8002ffd700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f800c11cf90 0x7f800c06dbf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:54.913 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.914 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.915 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:54.916 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:54.917 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:54.918 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:54.919 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:54.921 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.922 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:54.923 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.924 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.925 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:54.926 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:54.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 10 librados: watch_flush enter 2024-09-16T11:03:54.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 10 librados: watch_flush exit 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 20 client.84248.objecter shutdown clearing up homeless session... 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 10 client.84248.objecter successfully canceled tick 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7ff0036f60 msgr2=0x7f7ff0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 --2- 172.21.15.5:0/1940574044 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7ff0036f60 0x7f7ff0039420 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f80080097d0 tx=0x7f8008010000 comp rx=0 tx=0).stop 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 10 monclient: shutdown 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 msgr2=0x7f800c133100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 --2- 172.21.15.5:0/1940574044 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f800c09e8e0 0x7f800c133100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f800400cb90 tx=0x7f800400cea0 comp rx=0 tx=0).stop 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8012d98700 1 -- 172.21.15.5:0/1940574044 reap_dead start 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8012d98700 1 -- 172.21.15.5:0/1940574044 reap_dead start 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 shutdown_connections 2024-09-16T11:03:54.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 >> 172.21.15.5:0/1940574044 conn(0x7f800c0a45d0 msgr2=0x7f800c06cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:54.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 shutdown_connections 2024-09-16T11:03:54.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 -- 172.21.15.5:0/1940574044 wait complete. 2024-09-16T11:03:54.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:54.895+0000 7f8013599700 1 librados: shutdown 2024-09-16T11:03:54.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:54.930 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T11:03:54.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: get_monmap_and_config 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: build_initial_monmap 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: monmap: 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:55.049071+0000 2024-09-16T11:03:55.050 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:55.049071+0000 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:55.051 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 Processor -- start 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 -- start start 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: init 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a0135c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-b con 0x7f01a00a3990 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01a00a2270 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f01a00a2650 0x7f01a01329f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-a con 0x7f01a00a2650 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:55.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a0135890 con 0x7f01a00a2270 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a00a4aa0 con 0x7f01a00a2650 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00a3d70 con 0x7f01a00a3990 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient(hunting): authenticate will time out at 1693.626162s 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019f7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f01a00a2270 0x7f01a01322e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49508/0 (socket says 172.21.15.5:49508) 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a0135c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f01a00a2650 0x7f01a01329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019f7fe700 1 -- 172.21.15.5:0/1240364154 learned_addr learned my addr 172.21.15.5:0/1240364154 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): get_auth_request con 0x7f01a00a3990 auth_method 0 2024-09-16T11:03:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): get_auth_request con 0x7f01a00a2650 auth_method 0 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:55.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient(hunting): handle_auth_done global_id 84251 payload 274 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 -- 172.21.15.5:0/1240364154 >> v2:172.21.15.5:3300/0 conn(0x7f01a00a2650 msgr2=0x7f01a01329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 --2- 172.21.15.5:0/1240364154 >> v2:172.21.15.5:3300/0 conn(0x7f01a00a2650 0x7f01a01329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 -- 172.21.15.5:0/1240364154 >> v1:172.21.15.5:6789/0 conn(0x7f01a00a2270 legacy=0x7f01a01322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 monclient: found mon.noname-b 2024-09-16T11:03:55.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 -- 172.21.15.5:0/1240364154 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a009fa50 con 0x7f01a00a3990 2024-09-16T11:03:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019ffff700 1 --2- 172.21.15.5:0/1240364154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a0135c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0190009850 tx=0x7f0190009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3023631485eed61a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:55.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 1 -- 172.21.15.5:0/1240364154 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190011830 con 0x7f01a00a3990 2024-09-16T11:03:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: dump: 2024-09-16T11:03:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:55.059 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: _finish_auth 0 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:55.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 5 monclient: authenticate success, global_id 84251 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 1 -- 172.21.15.5:0/1240364154 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190011990 con 0x7f01a00a3990 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f01a661f700 4 set_mon_vals no callback set 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 1 -- 172.21.15.5:0/1240364154 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190011d40 con 0x7f01a00a3990 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.047+0000 7f019effd700 10 monclient: dump: 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.061 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:55.062 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: shutdown 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- 172.21.15.5:0/1240364154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 msgr2=0x7f01a0135c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 --2- 172.21.15.5:0/1240364154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a0135c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0190009850 tx=0x7f0190009b60 comp rx=0 tx=0).stop 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- 172.21.15.5:0/1240364154 shutdown_connections 2024-09-16T11:03:55.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 --2- 172.21.15.5:0/1240364154 >> v2:172.21.15.5:3300/0 conn(0x7f01a00a2650 0x7f01a01329f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 --2- 172.21.15.5:0/1240364154 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a0135c80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- 172.21.15.5:0/1240364154 >> 172.21.15.5:0/1240364154 conn(0x7f01a00a84b0 msgr2=0x7f01a00a88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- 172.21.15.5:0/1240364154 shutdown_connections 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- 172.21.15.5:0/1240364154 wait complete. 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: build_initial_monmap 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: monmap: 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:55.070 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 librados: starting msgr at 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 librados: starting objecter 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 Processor -- start 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 -- start start 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 librados: setting wanted keys 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 librados: calling monclient init 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: init 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:03:55.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 0x7f01a01ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-b con 0x7f01a00a2270 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.051+0000 7f01a661f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 0x7f01a01caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-a con 0x7f01a00a2650 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): picked mon.noname-c con 0x7f01a00a3990 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0070f10 con 0x7f01a00a2650 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a01201f0 con 0x7f01a00a2270 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0071420 con 0x7f01a00a3990 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:55.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient(hunting): authenticate will time out at 1693.632086s 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 0x7f01a01caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47688/0 (socket says 172.21.15.5:47688) 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 0x7f01a01caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57906/0 (socket says 172.21.15.5:57906) 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 -- 172.21.15.5:0/1527138269 learned_addr learned my addr 172.21.15.5:0/1527138269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): get_auth_request con 0x7f01a00a3990 auth_method 0 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:55.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): get_auth_request con 0x7f01a00a2650 auth_method 0 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 0x7f01a01ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 10 monclient(hunting): get_auth_request con 0x7f01a00a2270 auth_method 0 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:55.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient(hunting): handle_auth_done global_id 84254 payload 824 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 -- 172.21.15.5:0/1527138269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 msgr2=0x7f01a01ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 0x7f01a01ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 -- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 msgr2=0x7f01a01caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 0x7f01a01caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient: _finish_hunting 0 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 monclient: found mon.noname-c 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:55.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 -- 172.21.15.5:0/1527138269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900094b0 con 0x7f01a00a3990 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019ffff700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 0x7f01a01ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a4bbc700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0198009dd0 tx=0x7f019800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 client.?.objecter ms_handle_connect 0x7f01a00a3990 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198009c40 con 0x7f01a00a3990 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: dump: 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:55.076 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: _finish_auth 0 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: _check_auth_tickets 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 5 monclient: authenticate success, global_id 84254 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800f040 con 0x7f01a00a3990 2024-09-16T11:03:55.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _renew_subs 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a01204d0 con 0x7f01a00a3990 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198014480 con 0x7f01a00a3990 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 monclient: dump: 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:55.078 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019e7fc700 4 set_mon_vals no callback set 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 client.84254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _renew_subs 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00a1cc0 con 0x7f01a00a3990 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 librados: init done 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f01980145e0 con 0x7f01a00a3990 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a009f6a0 con 0x7f01a00a3990 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 --2- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 0x7f0180039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(572..572 src has 395..572) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f0198048ad0 con 0x7f01a00a3990 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 10 client.84254.objecter ms_dispatch 0x7f01a0000fd0 osd_map(572..572 src has 395..572) v4 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 3 client.84254.objecter handle_osd_map got epochs [572,572] > 0 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 3 client.84254.objecter handle_osd_map decoding full epoch 572 2024-09-16T11:03:55.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.055+0000 7f019cff9700 20 client.84254.objecter dump_active .. 0 homeless 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.059+0000 7f019ffff700 1 --2- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 0x7f0180039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.059+0000 7f019ffff700 10 monclient: get_auth_request con 0x7f0180036f90 auth_method 0 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.059+0000 7f019ffff700 1 --2- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 0x7f0180039450 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7f01900090b0 tx=0x7f0190008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.059+0000 7f019cff9700 10 client.84254.objecter ms_handle_connect 0x7f0180036f90 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.063+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0198007900 con 0x7f01a00a3990 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.063+0000 7f019cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:55.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.063+0000 7f019cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:55.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.311+0000 7f01a661f700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T11:03:55.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.311+0000 7f01a661f700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T11:03:55.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.311+0000 7f01a661f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:55.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.311+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f01a0135c60 con 0x7f01a00a3990 2024-09-16T11:03:55.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=3649194563,config=692619927,health=1904779523,kv=564032378,logm=2187685654}) 2024-09-16T11:03:55.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=595690931,config=692619927,health=1904779523,kv=3549593220,logm=912719783}) 2024-09-16T11:03:55.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f019cff9700 1 -- 172.21.15.5:0/1527138269 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0198018070 con 0x7f01a00a3990 2024-09-16T11:03:55.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f019cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T11:03:55.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f019cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:55.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=3649194563,config=692619927,health=1904779523,kv=564032378,logm=2187685654}) 2024-09-16T11:03:55.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.315+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=395387798,config=692619927,health=1904779523,kv=564032378,logm=927542506}) 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 10 librados: watch_flush enter 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 10 librados: watch_flush exit 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 20 client.84254.objecter shutdown clearing up homeless session... 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 10 client.84254.objecter successfully canceled tick 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 msgr2=0x7f0180039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 0x7f0180039450 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7f01900090b0 tx=0x7f0190008030 comp rx=0 tx=0).stop 2024-09-16T11:03:55.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 10 monclient: shutdown 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 msgr2=0x7f01a006f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0198009dd0 tx=0x7f019800c3a0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 shutdown_connections 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f01a00a3990 0x7f01a006f270 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0180036f90 0x7f0180039450 unknown :-1 s=CLOSED pgs=3868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01a00a2650 0x7f01a01caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 --2- 172.21.15.5:0/1527138269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01a00a2270 0x7f01a01ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 >> 172.21.15.5:0/1527138269 conn(0x7f01a00a84b0 msgr2=0x7f01a01321f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 shutdown_connections 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 -- 172.21.15.5:0/1527138269 wait complete. 2024-09-16T11:03:55.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f01a661f700 1 librados: shutdown 2024-09-16T11:03:55.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748158368}) 2024-09-16T11:03:55.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942446701}) 2024-09-16T11:03:55.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748158368}) 2024-09-16T11:03:55.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.319+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866111261}) 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576408629}) 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105595199}) 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576408629}) 2024-09-16T11:03:55.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.323+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1829155924}) 2024-09-16T11:03:55.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159763930}) 2024-09-16T11:03:55.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905916166}) 2024-09-16T11:03:55.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159763930}) 2024-09-16T11:03:55.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294558945}) 2024-09-16T11:03:55.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806816851}) 2024-09-16T11:03:55.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711383566}) 2024-09-16T11:03:55.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806816851}) 2024-09-16T11:03:55.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.327+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217665980}) 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725818334}) 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453696266}) 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725818334}) 2024-09-16T11:03:55.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450910529}) 2024-09-16T11:03:55.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925239982}) 2024-09-16T11:03:55.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=379948899}) 2024-09-16T11:03:55.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2925239982}) 2024-09-16T11:03:55.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.331+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068794661}) 2024-09-16T11:03:55.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T11:03:55.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271169642}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90155291}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271169642}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264497601}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166504175}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093664646}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166504175}) 2024-09-16T11:03:55.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396481506}) 2024-09-16T11:03:55.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.342 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122424103}) 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953277176}) 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122424103}) 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.335+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763938186}) 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.343 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825683717}) 2024-09-16T11:03:55.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844585004}) 2024-09-16T11:03:55.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825683717}) 2024-09-16T11:03:55.344 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083781480}) 2024-09-16T11:03:55.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979858417}) 2024-09-16T11:03:55.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181836277}) 2024-09-16T11:03:55.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979858417}) 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897513444}) 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670630763}) 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340099281}) 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670630763}) 2024-09-16T11:03:55.346 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.339+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284873808}) 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378262238}) 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190636572}) 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3378262238}) 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832685563}) 2024-09-16T11:03:55.347 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777019012}) 2024-09-16T11:03:55.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050189659}) 2024-09-16T11:03:55.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777019012}) 2024-09-16T11:03:55.348 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277022899}) 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596869022}) 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118656381}) 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596869022}) 2024-09-16T11:03:55.349 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.343+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=64597706}) 2024-09-16T11:03:55.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998288409}) 2024-09-16T11:03:55.350 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548875938}) 2024-09-16T11:03:55.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998288409}) 2024-09-16T11:03:55.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051570849}) 2024-09-16T11:03:55.351 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950822366}) 2024-09-16T11:03:55.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4020052962}) 2024-09-16T11:03:55.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950822366}) 2024-09-16T11:03:55.352 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358029045}) 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250802449}) 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387448072}) 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250802449}) 2024-09-16T11:03:55.353 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.347+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941926387}) 2024-09-16T11:03:55.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090437574}) 2024-09-16T11:03:55.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311372857}) 2024-09-16T11:03:55.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.354 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090437574}) 2024-09-16T11:03:55.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507392510}) 2024-09-16T11:03:55.355 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910221987}) 2024-09-16T11:03:55.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996937060}) 2024-09-16T11:03:55.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910221987}) 2024-09-16T11:03:55.356 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352386754}) 2024-09-16T11:03:55.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087449472}) 2024-09-16T11:03:55.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969306289}) 2024-09-16T11:03:55.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.357 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087449472}) 2024-09-16T11:03:55.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.351+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1711703309}) 2024-09-16T11:03:55.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680106251}) 2024-09-16T11:03:55.358 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526873132}) 2024-09-16T11:03:55.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680106251}) 2024-09-16T11:03:55.359 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433040730}) 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150011179}) 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535590573}) 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2150011179}) 2024-09-16T11:03:55.361 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.355+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938924112}) 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873528699}) 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983170038}) 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2873528699}) 2024-09-16T11:03:55.362 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2170601811}) 2024-09-16T11:03:55.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.363 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176999694}) 2024-09-16T11:03:55.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028636972}) 2024-09-16T11:03:55.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176999694}) 2024-09-16T11:03:55.364 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403913497}) 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871441265}) 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256554781}) 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871441265}) 2024-09-16T11:03:55.365 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.359+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551149107}) 2024-09-16T11:03:55.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700967251}) 2024-09-16T11:03:55.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752022916}) 2024-09-16T11:03:55.366 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1700967251}) 2024-09-16T11:03:55.367 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737506694}) 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709706027}) 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=495353059}) 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709706027}) 2024-09-16T11:03:55.368 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990845337}) 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653739410}) 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745565102}) 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653739410}) 2024-09-16T11:03:55.369 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.363+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272177666}) 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243604214}) 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018626575}) 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243604214}) 2024-09-16T11:03:55.370 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187620793}) 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606130342}) 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3002308816}) 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606130342}) 2024-09-16T11:03:55.372 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558293172}) 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954155628}) 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773972837}) 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954155628}) 2024-09-16T11:03:55.373 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.367+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3690241342}) 2024-09-16T11:03:55.374 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.374 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776378190}) 2024-09-16T11:03:55.374 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061174577}) 2024-09-16T11:03:55.374 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776378190}) 2024-09-16T11:03:55.375 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200164821}) 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836229137}) 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402998938}) 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836229137}) 2024-09-16T11:03:55.376 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207663408}) 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522004511}) 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737034193}) 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522004511}) 2024-09-16T11:03:55.377 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.371+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889225756}) 2024-09-16T11:03:55.380 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.380 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269777783}) 2024-09-16T11:03:55.380 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57492281}) 2024-09-16T11:03:55.380 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.380 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269777783}) 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473003127}) 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272879298}) 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1632567792}) 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272879298}) 2024-09-16T11:03:55.381 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588573070}) 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373731190}) 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248402896}) 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373731190}) 2024-09-16T11:03:55.382 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.375+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549145642}) 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039341394}) 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493193947}) 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3039341394}) 2024-09-16T11:03:55.383 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772590634}) 2024-09-16T11:03:55.384 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.384 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265607931}) 2024-09-16T11:03:55.385 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122109969}) 2024-09-16T11:03:55.385 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.385 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265607931}) 2024-09-16T11:03:55.385 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.379+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645863858}) 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029127847}) 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3789759764}) 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029127847}) 2024-09-16T11:03:55.386 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752384126}) 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052225465}) 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238095057}) 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052225465}) 2024-09-16T11:03:55.388 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1062634729}) 2024-09-16T11:03:55.389 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.389 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1103598057}) 2024-09-16T11:03:55.389 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497719701}) 2024-09-16T11:03:55.390 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.390 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1103598057}) 2024-09-16T11:03:55.390 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.383+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=580739611}) 2024-09-16T11:03:55.391 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.391 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357985863}) 2024-09-16T11:03:55.391 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136254165}) 2024-09-16T11:03:55.391 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.392 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357985863}) 2024-09-16T11:03:55.392 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465639739}) 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982545421}) 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490175347}) 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982545421}) 2024-09-16T11:03:55.393 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.387+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1192852133}) 2024-09-16T11:03:55.394 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.395 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182173895}) 2024-09-16T11:03:55.395 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893362286}) 2024-09-16T11:03:55.395 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.395 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182173895}) 2024-09-16T11:03:55.395 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805689543}) 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010181261}) 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536550333}) 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010181261}) 2024-09-16T11:03:55.396 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189933744}) 2024-09-16T11:03:55.397 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.397 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720174181}) 2024-09-16T11:03:55.397 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2832712133}) 2024-09-16T11:03:55.397 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720174181}) 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.391+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005012297}) 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941879915}) 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376967362}) 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.398 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3941879915}) 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743032825}) 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111515049}) 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118181819}) 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.399 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111515049}) 2024-09-16T11:03:55.400 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3632419466}) 2024-09-16T11:03:55.400 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.400 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950192060}) 2024-09-16T11:03:55.400 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676106123}) 2024-09-16T11:03:55.400 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950192060}) 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832965538}) 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746021145}) 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985255457}) 2024-09-16T11:03:55.401 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.402 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746021145}) 2024-09-16T11:03:55.402 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.395+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=99424207}) 2024-09-16T11:03:55.402 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.402 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=3,osd_metadata=3} crc {logm=1270031937,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1047873171,mon_config_key=3101995378,monmap=2357222013,osd_metadata=532213110}) 2024-09-16T11:03:55.402 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=2} crc {logm=447642455,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=250281147,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3538871177}) 2024-09-16T11:03:55.403 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.403 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=3,osd_metadata=3} crc {logm=1270031937,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1047873171,mon_config_key=3101995378,monmap=2357222013,osd_metadata=532213110}) 2024-09-16T11:03:55.403 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=3} crc {logm=3253785385,mdsmap=1531151339,mgr=2079724239,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=3726879211,mon_config_key=3273021363,monmap=2357222013,osd_metadata=532213110}) 2024-09-16T11:03:55.403 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.403 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=44,osdmap=52} crc {osd_metadata=3535119589,osd_pg_creating=3373043507,osd_snap=3301680806,osdmap=3360752764}) 2024-09-16T11:03:55.404 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=49,osdmap=47} crc {osd_metadata=3535119589,osd_pg_creating=3373043507,osd_snap=245895740,osdmap=461000246}) 2024-09-16T11:03:55.404 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.404 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=44,osdmap=52} crc {osd_metadata=3535119589,osd_pg_creating=3373043507,osd_snap=3301680806,osdmap=3360752764}) 2024-09-16T11:03:55.404 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=46,osdmap=50} crc {osd_metadata=3535119589,osd_pg_creating=3373043507,osd_snap=3175725253,osdmap=590718452}) 2024-09-16T11:03:55.404 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.405 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1871879914}) 2024-09-16T11:03:55.405 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3083585576}) 2024-09-16T11:03:55.405 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.405 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1871879914}) 2024-09-16T11:03:55.405 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.399+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3305433893}) 2024-09-16T11:03:55.406 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.406 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1342256299}) 2024-09-16T11:03:55.406 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4099193207}) 2024-09-16T11:03:55.406 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.406 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1342256299}) 2024-09-16T11:03:55.407 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.403+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3524424041}) 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=3625990390}) 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=1469881759}) 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=3625990390}) 2024-09-16T11:03:55.411 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:03:55.407+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=4103552468}) 2024-09-16T11:03:56.340 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:03:56.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:56.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.451+0000 7f105359e700 10 monclient: get_monmap_and_config 2024-09-16T11:03:56.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.451+0000 7f105359e700 10 monclient: build_initial_monmap 2024-09-16T11:03:56.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.451+0000 7f105359e700 10 monclient: monmap: 2024-09-16T11:03:56.457 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:56.456150+0000 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:56.456150+0000 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:56.458 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 Processor -- start 2024-09-16T11:03:56.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- start start 2024-09-16T11:03:56.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient: init 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10540a2c80 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10540a3770 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 0x7f105413e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): picked mon.noname-b con 0x7f105413c3a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540a3db0 con 0x7f10540a2c80 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540a4b70 con 0x7f10540a3770 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540a0850 con 0x7f105413c3a0 2024-09-16T11:03:56.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient(hunting): authenticate will time out at 1695.033075s 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105259c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f10540a2c80 0x7f10540a3060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49524/0 (socket says 172.21.15.5:49524) 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105259c700 1 -- 172.21.15.5:0/1386664166 learned_addr learned my addr 172.21.15.5:0/1386664166 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 --2- 172.21.15.5:0/1386664166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 0x7f105413e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): get_auth_request con 0x7f105413c3a0 auth_method 0 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient(hunting): handle_auth_done global_id 84257 payload 274 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 1 -- 172.21.15.5:0/1386664166 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307075909 0 0) 0x7f10540a3db0 con 0x7f10540a2c80 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 -- 172.21.15.5:0/1386664166 >> v1:172.21.15.143:6789/0 conn(0x7f10540a3770 legacy=0x7f105413ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 -- 172.21.15.5:0/1386664166 >> v1:172.21.15.5:6789/0 conn(0x7f10540a2c80 legacy=0x7f10540a3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 monclient: found mon.noname-b 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 -- 172.21.15.5:0/1386664166 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540a4180 con 0x7f105413c3a0 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1052d9d700 1 --2- 172.21.15.5:0/1386664166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 0x7f105413e790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f104c009dc0 tx=0x7f104c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3b1ed7c9e08595 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 1 -- 172.21.15.5:0/1386664166 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c00e070 con 0x7f105413c3a0 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: dump: 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: _finish_auth 0 2024-09-16T11:03:56.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: _check_auth_tickets 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 5 monclient: authenticate success, global_id 84257 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 1 -- 172.21.15.5:0/1386664166 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c00bbd0 con 0x7f105413c3a0 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 4 set_mon_vals no callback set 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 1 -- 172.21.15.5:0/1386664166 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c012430 con 0x7f105413c3a0 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f1051d9b700 10 monclient: dump: 2024-09-16T11:03:56.467 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 10 monclient: shutdown 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- 172.21.15.5:0/1386664166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 msgr2=0x7f105413e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 --2- 172.21.15.5:0/1386664166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 0x7f105413e790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f104c009dc0 tx=0x7f104c00b330 comp rx=0 tx=0).stop 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- 172.21.15.5:0/1386664166 shutdown_connections 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 --2- 172.21.15.5:0/1386664166 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f105413c3a0 0x7f105413e790 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- 172.21.15.5:0/1386664166 >> 172.21.15.5:0/1386664166 conn(0x7f10540a4d80 msgr2=0x7f10540a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- 172.21.15.5:0/1386664166 shutdown_connections 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.455+0000 7f105359e700 1 -- 172.21.15.5:0/1386664166 wait complete. 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: build_initial_monmap 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: monmap: 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:56.469 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 librados: starting msgr at 2024-09-16T11:03:56.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 librados: starting objecter 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 Processor -- start 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- start start 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 librados: setting wanted keys 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 librados: calling monclient init 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: init 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:03:56.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10540a2c80 0x7f1054134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): picked mon.noname-b con 0x7f10540a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10540a3770 0x7f1054134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): picked mon.noname-c con 0x7f10540a3770 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): picked mon.noname-a con 0x7f105413c3a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540a7490 con 0x7f105413c3a0 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1054120410 con 0x7f10540a2c80 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540706b0 con 0x7f10540a3770 2024-09-16T11:03:56.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient(hunting): authenticate will time out at 1695.038007s 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57920/0 (socket says 172.21.15.5:57920) 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 -- 172.21.15.5:0/1417169718 learned_addr learned my addr 172.21.15.5:0/1417169718 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): get_auth_request con 0x7f105413c3a0 auth_method 0 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10540a2c80 0x7f1054134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10540a3770 0x7f1054134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): get_auth_request con 0x7f10540a2c80 auth_method 0 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): get_auth_request con 0x7f10540a3770 auth_method 0 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient(hunting): handle_auth_done global_id 74433 payload 824 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 -- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10540a3770 msgr2=0x7f1054134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10540a3770 0x7f1054134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 -- 172.21.15.5:0/1417169718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10540a2c80 msgr2=0x7f1054134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10540a2c80 0x7f1054134a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 monclient: found mon.noname-a 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 -- 172.21.15.5:0/1417169718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c009a70 con 0x7f105413c3a0 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 client.?.objecter ms_handle_connect 0x7f105413c3a0 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f1052d9d700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f104c009f30 tx=0x7f104c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c01b8d0 con 0x7f105413c3a0 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: dump: 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: _finish_auth 0 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 5 monclient: authenticate success, global_id 74433 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _renew_subs 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c020070 con 0x7f105413c3a0 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1054135a40 con 0x7f105413c3a0 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c01bcb0 con 0x7f105413c3a0 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f104affd700 10 monclient: dump: 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105159a700 4 set_mon_vals no callback set 2024-09-16T11:03:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 client.74433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _renew_subs 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1054070190 con 0x7f105413c3a0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.459+0000 7f105359e700 1 librados: init done 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f104c01c400 con 0x7f105413c3a0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 1 --2- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 0x7f1034039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10540a3060 con 0x7f105413c3a0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(574..574 src has 395..574) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f104c022070 con 0x7f105413c3a0 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 10 client.74433.objecter ms_dispatch 0x7f1054000fd0 osd_map(574..574 src has 395..574) v4 2024-09-16T11:03:56.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 3 client.74433.objecter handle_osd_map got epochs [574,574] > 0 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 3 client.74433.objecter handle_osd_map decoding full epoch 574 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 20 client.74433.objecter dump_active .. 0 homeless 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105259c700 1 --2- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 0x7f1034039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105259c700 10 monclient: get_auth_request con 0x7f1034036f60 auth_method 0 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f104affd700 10 client.74433.objecter ms_handle_connect 0x7f1034036f60 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.463+0000 7f105259c700 1 --2- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 0x7f1034039420 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7f1044000f60 tx=0x7f1044004a70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.471+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f104c02c070 con 0x7f105413c3a0 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.471+0000 7f104affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.471+0000 7f104affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:56.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f105359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:56.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f105359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:56.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f105359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:56.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f105411fd40 con 0x7f105413c3a0 2024-09-16T11:03:56.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f104affd700 1 -- 172.21.15.5:0/1417169718 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f104c01be10 con 0x7f105413c3a0 2024-09-16T11:03:56.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f104affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:56.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.711+0000 7f104affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:56.716 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:56.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 10 librados: watch_flush enter 2024-09-16T11:03:56.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 10 librados: watch_flush exit 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 20 client.74433.objecter shutdown clearing up homeless session... 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 10 client.74433.objecter successfully canceled tick 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 msgr2=0x7f1034039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 0x7f1034039420 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7f1044000f60 tx=0x7f1044004a70 comp rx=0 tx=0).stop 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 10 monclient: shutdown 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 msgr2=0x7f105406e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f104c009f30 tx=0x7f104c004af0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 shutdown_connections 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1034036f60 0x7f1034039420 unknown :-1 s=CLOSED pgs=3869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f105413c3a0 0x7f105406e5e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f10540a3770 0x7f1054134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 --2- 172.21.15.5:0/1417169718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10540a2c80 0x7f1054134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 >> 172.21.15.5:0/1417169718 conn(0x7f10540a4d80 msgr2=0x7f1054138410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 shutdown_connections 2024-09-16T11:03:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 -- 172.21.15.5:0/1417169718 wait complete. 2024-09-16T11:03:56.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.715+0000 7f105359e700 1 librados: shutdown 2024-09-16T11:03:56.730 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:03:56.731 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T11:03:56.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:03:56.731 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.835+0000 7f0894a2b700 10 monclient: get_monmap_and_config 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.835+0000 7f0894a2b700 10 monclient: build_initial_monmap 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.835+0000 7f0894a2b700 10 monclient: monmap: 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:56.840036+0000 2024-09-16T11:03:56.841 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:56.840036+0000 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:56.842 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 Processor -- start 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- start start 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient: init 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 0x7f08900a3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-b con 0x7f08900a2c80 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:56.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08900a35a0 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0890136150 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08900a3db0 con 0x7f08900a35a0 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08900a4b70 con 0x7f0890136150 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08900a0850 con 0x7f08900a2c80 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient(hunting): authenticate will time out at 1695.417143s 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088659c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f08900a35a0 0x7f0890135a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49532/0 (socket says 172.21.15.5:49532) 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088659c700 1 -- 172.21.15.5:0/1751766155 learned_addr learned my addr 172.21.15.5:0/1751766155 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 --2- 172.21.15.5:0/1751766155 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 0x7f08900a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): get_auth_request con 0x7f08900a2c80 auth_method 0 2024-09-16T11:03:56.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 1 -- 172.21.15.5:0/1751766155 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246124358 0 0) 0x7f08900a3db0 con 0x7f08900a35a0 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient(hunting): _negotiate 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient(hunting): my global_id is 74436 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 1 -- 172.21.15.5:0/1751766155 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c0035d0 con 0x7f08900a35a0 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient(hunting): handle_auth_done global_id 84263 payload 274 2024-09-16T11:03:56.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 -- 172.21.15.5:0/1751766155 >> v1:172.21.15.143:6789/0 conn(0x7f0890136150 legacy=0x7f089013dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 -- 172.21.15.5:0/1751766155 >> v1:172.21.15.5:6789/0 conn(0x7f08900a35a0 legacy=0x7f0890135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient: _finish_hunting 0 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 monclient: found mon.noname-b 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 -- 172.21.15.5:0/1751766155 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08900a4180 con 0x7f08900a2c80 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088ed9d700 1 --2- 172.21.15.5:0/1751766155 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 0x7f08900a3060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0880009880 tx=0x7f0880009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b55d5b0b32a6a64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 1 -- 172.21.15.5:0/1751766155 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880011830 con 0x7f08900a2c80 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: dump: 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.849 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: _finish_auth 0 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: _check_auth_tickets 2024-09-16T11:03:56.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 1 -- 172.21.15.5:0/1751766155 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0880011990 con 0x7f08900a2c80 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 5 monclient: authenticate success, global_id 84263 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 1 -- 172.21.15.5:0/1751766155 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880011d20 con 0x7f08900a2c80 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 4 set_mon_vals no callback set 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f088e59c700 10 monclient: dump: 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.851 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 10 monclient: shutdown 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- 172.21.15.5:0/1751766155 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 msgr2=0x7f08900a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 --2- 172.21.15.5:0/1751766155 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 0x7f08900a3060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0880009880 tx=0x7f0880009b90 comp rx=0 tx=0).stop 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- 172.21.15.5:0/1751766155 shutdown_connections 2024-09-16T11:03:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 --2- 172.21.15.5:0/1751766155 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f08900a2c80 0x7f08900a3060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- 172.21.15.5:0/1751766155 >> 172.21.15.5:0/1751766155 conn(0x7f08900a4d80 msgr2=0x7f08900a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- 172.21.15.5:0/1751766155 shutdown_connections 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.839+0000 7f0894a2b700 1 -- 172.21.15.5:0/1751766155 wait complete. 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: build_initial_monmap 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: monmap: 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:56.854 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 librados: starting msgr at 2024-09-16T11:03:56.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 librados: starting objecter 2024-09-16T11:03:56.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 Processor -- start 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- start start 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 librados: setting wanted keys 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 librados: calling monclient init 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: init 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08900a2c80 0x7f08901ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-b con 0x7f08900a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08900a35a0 0x7f0890134af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-a con 0x7f08900a35a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): picked mon.noname-c con 0x7f0890136150 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890126030 con 0x7f08900a35a0 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08901ca930 con 0x7f08900a2c80 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0890135a20 con 0x7f0890136150 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient(hunting): authenticate will time out at 1695.421806s 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08900a35a0 0x7f0890134af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47734/0 (socket says 172.21.15.5:47734) 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 -- 172.21.15.5:0/961898459 learned_addr learned my addr 172.21.15.5:0/961898459 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:56.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): get_auth_request con 0x7f0890136150 auth_method 0 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): get_auth_request con 0x7f08900a35a0 auth_method 0 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08900a2c80 0x7f08901ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): get_auth_request con 0x7f08900a2c80 auth_method 0 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient(hunting): handle_auth_done global_id 84266 payload 824 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08900a2c80 msgr2=0x7f08901ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08900a2c80 0x7f08901ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08900a35a0 msgr2=0x7f0890134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08900a35a0 0x7f0890134af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient: _finish_hunting 0 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 monclient: found mon.noname-c 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800094b0 con 0x7f0890136150 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 client.?.objecter ms_handle_connect 0x7f0890136150 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088f59e700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f088800cbd0 tx=0x7f088800cee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888003c40 con 0x7f0890136150 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: dump: 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: _finish_auth 0 2024-09-16T11:03:56.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: _check_auth_tickets 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0888003da0 con 0x7f0890136150 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 5 monclient: authenticate success, global_id 84266 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _renew_subs 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08900a7240 con 0x7f0890136150 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088800aae0 con 0x7f0890136150 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0887fff700 10 monclient: dump: 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.867 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f088dd9b700 4 set_mon_vals no callback set 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 client.84266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _renew_subs 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0890126210 con 0x7f0890136150 2024-09-16T11:03:56.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.843+0000 7f0894a2b700 1 librados: init done 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0894a2b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0894a2b700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 0x7f08901228d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890067380 con 0x7f08901224f0 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 0x7f08901228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient: get_auth_request con 0x7f08901224f0 auth_method 0 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f088800ac40 con 0x7f0890136150 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 10 monclient(hunting): handle_auth_done global_id 74442 payload 824 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f089009fa10 con 0x7f08901224f0 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 --2- 172.21.15.5:0/961898459 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0870036f90 0x7f0870039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088ed9d700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 0x7f08901228d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0880009880 tx=0x7f0880004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(574..574 src has 395..574) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f0888047f40 con 0x7f0890136150 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 client.84266.objecter ms_dispatch 0x7f0890000fd0 osd_map(574..574 src has 395..574) v4 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 3 client.84266.objecter handle_osd_map got epochs [574,574] > 0 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 3 client.84266.objecter handle_osd_map decoding full epoch 574 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088659c700 1 --2- 172.21.15.5:0/961898459 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0870036f90 0x7f0870039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088659c700 10 monclient: get_auth_request con 0x7f0870036f90 auth_method 0 2024-09-16T11:03:56.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 20 client.84266.objecter dump_active .. 0 homeless 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 client.84266.objecter ms_handle_connect 0x7f08901224f0 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 client.84266.objecter resend_mon_ops 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880004570 con 0x7f08901224f0 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 monclient: dump: 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.871 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f08800036c0 con 0x7f08901224f0 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 msgr2=0x7f08901228d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 0x7f08901228d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0880009880 tx=0x7f0880004770 comp rx=0 tx=0).stop 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 msgr2=0x7f08901228d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08901224f0 0x7f08901228d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f088659c700 1 --2- 172.21.15.5:0/961898459 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0870036f90 0x7f0870039450 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7f0890123d90 tx=0x7f0878007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.847+0000 7f0887fff700 10 client.84266.objecter ms_handle_connect 0x7f0870036f90 2024-09-16T11:03:56.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f0894a2b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:56.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f0894a2b700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 0x7f089011e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:56.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f0894a2b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:56.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890124fe0 con 0x7f0890122e10 2024-09-16T11:03:56.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 0x7f089011e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient: get_auth_request con 0x7f0890122e10 auth_method 0 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.855+0000 7f088ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f088ed9d700 10 monclient(hunting): handle_auth_done global_id 74445 payload 824 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f088ed9d700 1 -- 172.21.15.5:0/961898459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08900a4400 con 0x7f0890122e10 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f088ed9d700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 0x7f089011e8d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08800097d0 tx=0x7f0880008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 client.84266.objecter ms_handle_connect 0x7f0890122e10 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 client.84266.objecter resend_mon_ops 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088002ab30 con 0x7f0890122e10 2024-09-16T11:03:56.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 monclient: dump: 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:56.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f088002ac90 con 0x7f0890122e10 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 msgr2=0x7f089011e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 0x7f089011e8d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f08800097d0 tx=0x7f0880008030 comp rx=0 tx=0).stop 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 msgr2=0x7f089011e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0887fff700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0890122e10 0x7f089011e8d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:56.877 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:56.878 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:56.879 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:56.880 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:56.881 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.882 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.883 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:56.884 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:56.885 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.886 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.887 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.888 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:56.889 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 10 librados: watch_flush enter 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 10 librados: watch_flush exit 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 20 client.84266.objecter shutdown clearing up homeless session... 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 10 client.84266.objecter successfully canceled tick 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0870036f90 msgr2=0x7f0870039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 --2- 172.21.15.5:0/961898459 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0870036f90 0x7f0870039450 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7f0890123d90 tx=0x7f0878007de0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 10 monclient: shutdown 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f088f59e700 1 -- 172.21.15.5:0/961898459 reap_dead start 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 msgr2=0x7f0890135030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f088800cbd0 tx=0x7f088800cee0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 shutdown_connections 2024-09-16T11:03:56.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 --2- 172.21.15.5:0/961898459 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f0890136150 0x7f0890135030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:56.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 >> 172.21.15.5:0/961898459 conn(0x7f08900a4d80 msgr2=0x7f08900a6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:56.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 shutdown_connections 2024-09-16T11:03:56.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 -- 172.21.15.5:0/961898459 wait complete. 2024-09-16T11:03:56.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:56.859+0000 7f0894a2b700 1 librados: shutdown 2024-09-16T11:03:56.892 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:56.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:03:56.892 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: get_monmap_and_config 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: monmap: 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:57.014875+0000 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:57.014875+0000 2024-09-16T11:03:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:57.017 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 Processor -- start 2024-09-16T11:03:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 -- start start 2024-09-16T11:03:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: init 2024-09-16T11:03:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f204809b3e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f204809bed0 0x7f20481378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-c con 0x7f204809bed0 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-a con 0x7f204809c2e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204809e350 con 0x7f204809b3e0 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204809ca20 con 0x7f204809c2e0 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204809cd00 con 0x7f204809bed0 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204d69f700 10 monclient(hunting): authenticate will time out at 1695.591329s 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57952/0 (socket says 172.21.15.5:57952) 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 1 -- 172.21.15.5:0/2841783300 learned_addr learned my addr 172.21.15.5:0/2841783300 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): get_auth_request con 0x7f204809c2e0 auth_method 0 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 1 --2- 172.21.15.5:0/2841783300 >> v2:172.21.15.143:3300/0 conn(0x7f204809bed0 0x7f20481378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): get_auth_request con 0x7f204809bed0 auth_method 0 2024-09-16T11:03:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.011+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 10 monclient(hunting): handle_auth_done global_id 94123 payload 274 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 -- 172.21.15.5:0/2841783300 >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 msgr2=0x7f2048139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 --2- 172.21.15.5:0/2841783300 >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 -- 172.21.15.5:0/2841783300 >> v1:172.21.15.143:6789/0 conn(0x7f204809b3e0 legacy=0x7f204809b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 monclient: found mon.noname-c 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 -- 172.21.15.5:0/2841783300 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204809cfe0 con 0x7f204809bed0 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204ce9e700 1 --2- 172.21.15.5:0/2841783300 >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f203ffff700 1 --2- 172.21.15.5:0/2841783300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481378f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2030009850 tx=0x7f2030009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e3da31b0848fb98 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 1 -- 172.21.15.5:0/2841783300 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000f8a0 con 0x7f204809bed0 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: dump: 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.024 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 5 monclient: authenticate success, global_id 94123 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 1 -- 172.21.15.5:0/2841783300 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203000fa00 con 0x7f204809bed0 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 1 -- 172.21.15.5:0/2841783300 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203000fd90 con 0x7f204809bed0 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 4 set_mon_vals no callback set 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f20477fe700 10 monclient: dump: 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.026 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 10 monclient: shutdown 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- 172.21.15.5:0/2841783300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 msgr2=0x7f20481378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 --2- 172.21.15.5:0/2841783300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481378f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2030009850 tx=0x7f2030009b60 comp rx=0 tx=0).stop 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- 172.21.15.5:0/2841783300 shutdown_connections 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 --2- 172.21.15.5:0/2841783300 >> v2:172.21.15.5:3300/0 conn(0x7f204809c2e0 0x7f2048139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 --2- 172.21.15.5:0/2841783300 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481378f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- 172.21.15.5:0/2841783300 >> 172.21.15.5:0/2841783300 conn(0x7f20480a45d0 msgr2=0x7f20480a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- 172.21.15.5:0/2841783300 shutdown_connections 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- 172.21.15.5:0/2841783300 wait complete. 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 10 monclient: monmap: 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:57.029 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 librados: starting msgr at 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 librados: starting objecter 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 Processor -- start 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 -- start start 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 librados: setting wanted keys 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 1 librados: calling monclient init 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.015+0000 7f204d69f700 10 monclient: init 2024-09-16T11:03:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204809b3e0 0x7f20481c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-c con 0x7f204809b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-b con 0x7f204809bed0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 0x7f20481c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): picked mon.noname-a con 0x7f204809c2e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20481a51f0 con 0x7f204809c2e0 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20481a7630 con 0x7f204809bed0 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20481a5860 con 0x7f204809b3e0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): authenticate will time out at 1695.596396s 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 0x7f20481c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204809b3e0 0x7f20481c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 0x7f20481c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57960/0 (socket says 172.21.15.5:57960) 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 1 -- 172.21.15.5:0/2283941106 learned_addr learned my addr 172.21.15.5:0/2283941106 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): get_auth_request con 0x7f204809c2e0 auth_method 0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): get_auth_request con 0x7f204809bed0 auth_method 0 2024-09-16T11:03:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): get_auth_request con 0x7f204809b3e0 auth_method 0 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient(hunting): handle_auth_done global_id 94129 payload 824 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204809b3e0 msgr2=0x7f20481c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204809b3e0 0x7f20481c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 msgr2=0x7f20481c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 0x7f20481c2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 monclient: found mon.noname-b 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2040009b30 con 0x7f204809bed0 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2047fff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204809b3e0 0x7f20481c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204ce9e700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f204809c2e0 0x7f20481c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f203ffff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481c22b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20480a1180 tx=0x7f2030004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 client.?.objecter ms_handle_connect 0x7f204809bed0 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203001d040 con 0x7f204809bed0 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: dump: 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.040 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20300038d0 con 0x7f204809bed0 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 5 monclient: authenticate success, global_id 94129 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _renew_subs 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030003c80 con 0x7f204809bed0 2024-09-16T11:03:57.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20300094b0 con 0x7f204809bed0 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f2046ffd700 4 set_mon_vals no callback set 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 10 monclient: dump: 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 client.94129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _renew_subs 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2048070fc0 con 0x7f204809bed0 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 librados: init done 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 0x7f204802b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20481a79d0 con 0x7f20481c27f0 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.019+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f203002c410 con 0x7f204809bed0 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 0x7f204802b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient: get_auth_request con 0x7f20481c27f0 auth_method 0 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 --2- 172.21.15.5:0/2283941106 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2028036f60 0x7f2028039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 10 monclient(hunting): handle_auth_done global_id 84272 payload 824 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f204809e8e0 con 0x7f20481c27f0 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(574..574 src has 395..574) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f203004b820 con 0x7f204809bed0 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 client.94129.objecter ms_dispatch 0x7f2048000fd0 osd_map(574..574 src has 395..574) v4 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 3 client.94129.objecter handle_osd_map got epochs [574,574] > 0 2024-09-16T11:03:57.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 3 client.94129.objecter handle_osd_map decoding full epoch 574 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f2047fff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 0x7f204802b6a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f203800faa0 tx=0x7f203800fe60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 20 client.94129.objecter dump_active .. 0 homeless 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 client.94129.objecter ms_handle_connect 0x7f20481c27f0 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 client.94129.objecter resend_mon_ops 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f204ce9e700 1 --2- 172.21.15.5:0/2283941106 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2028036f60 0x7f2028039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038017070 con 0x7f20481c27f0 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f204ce9e700 10 monclient: get_auth_request con 0x7f2028036f60 auth_method 0 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 monclient: dump: 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.045 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f20380052e0 con 0x7f20481c27f0 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 msgr2=0x7f204802b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 0x7f204802b6a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f203800faa0 tx=0x7f203800fe60 comp rx=0 tx=0).stop 2024-09-16T11:03:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 msgr2=0x7f204802b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f20481c27f0 0x7f204802b6a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f204ce9e700 1 --2- 172.21.15.5:0/2283941106 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2028036f60 0x7f2028039420 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f20481a63c0 tx=0x7f204000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.023+0000 7f20457fa700 10 client.94129.objecter ms_handle_connect 0x7f2028036f60 2024-09-16T11:03:57.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f204d69f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f204d69f700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 0x7f204802bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f204d69f700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2048067380 con 0x7f204802bbe0 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 0x7f204802bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient: get_auth_request con 0x7f204802bbe0 auth_method 0 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 10 monclient(hunting): handle_auth_done global_id 84275 payload 824 2024-09-16T11:03:57.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.031+0000 7f2047fff700 1 -- 172.21.15.5:0/2283941106 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f20481a0a00 con 0x7f204802bbe0 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 client.94129.objecter ms_handle_connect 0x7f204802bbe0 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 client.94129.objecter resend_mon_ops 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f2047fff700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 0x7f204802bff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f203800faa0 tx=0x7f2038007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038023070 con 0x7f204802bbe0 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 monclient: dump: 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.050 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4777 (secure 0 0 0) 0x7f203801ee10 con 0x7f204802bbe0 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 msgr2=0x7f204802bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 0x7f204802bff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f203800faa0 tx=0x7f2038007f90 comp rx=0 tx=0).stop 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 msgr2=0x7f204802bff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f20457fa700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f204802bbe0 0x7f204802bff0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:57.052 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.053 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.054 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:57.055 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.058 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.059 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.060 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.061 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.062 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.063 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:03:57.064 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:57.065 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 10 librados: watch_flush enter 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 10 librados: watch_flush exit 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 20 client.94129.objecter shutdown clearing up homeless session... 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 10 client.94129.objecter successfully canceled tick 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2028036f60 msgr2=0x7f2028039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 --2- 172.21.15.5:0/2283941106 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2028036f60 0x7f2028039420 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f20481a63c0 tx=0x7f204000b040 comp rx=0 tx=0).stop 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 10 monclient: shutdown 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 msgr2=0x7f20481c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 --2- 172.21.15.5:0/2283941106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f204809bed0 0x7f20481c22b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20480a1180 tx=0x7f2030004770 comp rx=0 tx=0).stop 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204ce9e700 1 -- 172.21.15.5:0/2283941106 reap_dead start 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204ce9e700 1 -- 172.21.15.5:0/2283941106 reap_dead start 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 shutdown_connections 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 >> 172.21.15.5:0/2283941106 conn(0x7f20480a45d0 msgr2=0x7f20480a26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 shutdown_connections 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 -- 172.21.15.5:0/2283941106 wait complete. 2024-09-16T11:03:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.035+0000 7f204d69f700 1 librados: shutdown 2024-09-16T11:03:57.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:57.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:03:57.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:03:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: get_monmap_and_config 2024-09-16T11:03:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: monmap: 2024-09-16T11:03:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:57.190279+0000 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:57.190279+0000 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:57.192 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 Processor -- start 2024-09-16T11:03:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 -- start start 2024-09-16T11:03:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: init 2024-09-16T11:03:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d7c071870 0x7f3d7c09aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-c con 0x7f3d7c071870 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d7c0a10f0 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d7c0a1540 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c09caa0 con 0x7f3d7c0a1540 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c09e3a0 con 0x7f3d7c0a10f0 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7c09f700 con 0x7f3d7c071870 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8103e700 10 monclient(hunting): authenticate will time out at 1695.766791s 2024-09-16T11:03:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8083d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3d7c0a1540 0x7f3d7c09eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49540/0 (socket says 172.21.15.5:49540) 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d8083d700 1 -- 172.21.15.5:0/2424077100 learned_addr learned my addr 172.21.15.5:0/2424077100 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 1 --2- 172.21.15.5:0/2424077100 >> v2:172.21.15.143:3300/0 conn(0x7f3d7c071870 0x7f3d7c09aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): get_auth_request con 0x7f3d7c071870 auth_method 0 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160819335 0 0) 0x7f3d74001490 con 0x7f3d7c0a1540 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _negotiate 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): my global_id is 74454 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d640035d0 con 0x7f3d7c0a1540 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135211146 0 0) 0x7f3d7c09e3a0 con 0x7f3d7c0a10f0 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _negotiate 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 10 monclient(hunting): my global_id is 94135 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d74001490 con 0x7f3d7c0a10f0 2024-09-16T11:03:57.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_done global_id 94132 payload 274 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 1 -- 172.21.15.5:0/2424077100 >> v1:172.21.15.143:6789/0 conn(0x7f3d7c0a10f0 legacy=0x7f3d7c09e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.187+0000 7f3d7bfff700 1 -- 172.21.15.5:0/2424077100 >> v1:172.21.15.5:6789/0 conn(0x7f3d7c0a1540 legacy=0x7f3d7c09eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646814373 0 0) 0x7f3d640035d0 con 0x7f3d7c0a1540 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 1 monclient: found mon.noname-c 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 1 -- 172.21.15.5:0/2424077100 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c09b5f0 con 0x7f3d7c071870 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7bfff700 1 --2- 172.21.15.5:0/2424077100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c071870 0x7f3d7c09aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009850 tx=0x7f3d70009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39e359c712a98733 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74001b80 con 0x7f3d7c071870 2024-09-16T11:03:57.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: dump: 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.201 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70004030 con 0x7f3d7c071870 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 1 -- 172.21.15.5:0/2424077100 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d700043c0 con 0x7f3d7c071870 2024-09-16T11:03:57.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d7affd700 10 monclient: dump: 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.203 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 5 monclient: authenticate success, global_id 94132 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 4 set_mon_vals no callback set 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 10 monclient: shutdown 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- 172.21.15.5:0/2424077100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c071870 msgr2=0x7f3d7c09aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 --2- 172.21.15.5:0/2424077100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c071870 0x7f3d7c09aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009850 tx=0x7f3d70009b60 comp rx=0 tx=0).stop 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- 172.21.15.5:0/2424077100 shutdown_connections 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 --2- 172.21.15.5:0/2424077100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c071870 0x7f3d7c09aa50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- 172.21.15.5:0/2424077100 >> 172.21.15.5:0/2424077100 conn(0x7f3d7c0a45d0 msgr2=0x7f3d7c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- 172.21.15.5:0/2424077100 shutdown_connections 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- 172.21.15.5:0/2424077100 wait complete. 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 10 monclient: monmap: 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:57.205 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:57.206 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:57.206 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 librados: starting msgr at 2024-09-16T11:03:57.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 librados: starting objecter 2024-09-16T11:03:57.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 Processor -- start 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 -- start start 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 librados: setting wanted keys 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 1 librados: calling monclient init 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.191+0000 7f3d8103e700 10 monclient: init 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 0x7f3d7c1c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-c con 0x7f3d7c071870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d7c0a10f0 0x7f3d7c1bffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-a con 0x7f3d7c0a10f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c0a1540 0x7f3d7c1c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): picked mon.noname-b con 0x7f3d7c0a1540 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c1a7090 con 0x7f3d7c0a10f0 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c1a4ea0 con 0x7f3d7c0a1540 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0a3610 con 0x7f3d7c071870 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): authenticate will time out at 1695.772273s 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 0x7f3d7c1c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 0x7f3d7c1c5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47778/0 (socket says 172.21.15.5:47778) 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 -- 172.21.15.5:0/1893789647 learned_addr learned my addr 172.21.15.5:0/1893789647 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): get_auth_request con 0x7f3d7c071870 auth_method 0 2024-09-16T11:03:57.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d7c0a10f0 0x7f3d7c1bffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): get_auth_request con 0x7f3d7c0a10f0 auth_method 0 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c0a1540 0x7f3d7c1c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 10 monclient(hunting): get_auth_request con 0x7f3d7c0a1540 auth_method 0 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient(hunting): handle_auth_done global_id 84278 payload 824 2024-09-16T11:03:57.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c0a1540 msgr2=0x7f3d7c1c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c0a1540 0x7f3d7c1c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d7c0a10f0 msgr2=0x7f3d7c1bffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d7c0a10f0 0x7f3d7c1bffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 monclient: found mon.noname-c 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700094b0 con 0x7f3d7c071870 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7bfff700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 0x7f3d7c1c5f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c075050 tx=0x7f3d70004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 client.?.objecter ms_handle_connect 0x7f3d7c071870 2024-09-16T11:03:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70011630 con 0x7f3d7c071870 2024-09-16T11:03:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:57.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: dump: 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7001f070 con 0x7f3d7c071870 2024-09-16T11:03:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 5 monclient: authenticate success, global_id 84278 2024-09-16T11:03:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _renew_subs 2024-09-16T11:03:57.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c070670 con 0x7f3d7c071870 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d700119d0 con 0x7f3d7c071870 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 monclient: dump: 2024-09-16T11:03:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.224 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7a7fc700 4 set_mon_vals no callback set 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 client.84278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _renew_subs 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c1a7930 con 0x7f3d7c071870 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 librados: init done 2024-09-16T11:03:57.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3d70011b30 con 0x7f3d7c071870 2024-09-16T11:03:57.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 0x7f3d7c1c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c067380 con 0x7f3d7c1c2f80 2024-09-16T11:03:57.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 --2- 172.21.15.5:0/1893789647 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d68036f60 0x7f3d68039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(574..574 src has 395..574) v4 ==== 5755+0+0 (secure 0 0 0) 0x7f3d70021070 con 0x7f3d7c071870 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 10 client.84278.objecter ms_dispatch 0x7f3d7c000fd0 osd_map(574..574 src has 395..574) v4 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 3 client.84278.objecter handle_osd_map got epochs [574,574] > 0 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 3 client.84278.objecter handle_osd_map decoding full epoch 574 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 0x7f3d7c1c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d78ff9700 20 client.84278.objecter dump_active .. 0 homeless 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient: get_auth_request con 0x7f3d7c1c2f80 auth_method 0 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 1 --2- 172.21.15.5:0/1893789647 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d68036f60 0x7f3d68039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.195+0000 7f3d8083d700 10 monclient: get_auth_request con 0x7f3d68036f60 auth_method 0 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_done global_id 94141 payload 824 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d7b7fe700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d7c09d580 con 0x7f3d7c1c2f80 2024-09-16T11:03:57.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d7b7fe700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 0x7f3d7c1c3360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c00fb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 client.84278.objecter ms_handle_connect 0x7f3d7c1c2f80 2024-09-16T11:03:57.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 client.84278.objecter resend_mon_ops 2024-09-16T11:03:57.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 client.84278.objecter ms_handle_connect 0x7f3d68036f60 2024-09-16T11:03:57.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d8083d700 1 --2- 172.21.15.5:0/1893789647 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d68036f60 0x7f3d68039420 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0a1950 tx=0x7f3d74002030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c005320 con 0x7f3d7c1c2f80 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 monclient: dump: 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.232 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.233 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.233 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.233 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.233 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.233 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f3d6c005480 con 0x7f3d7c1c2f80 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:57.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 msgr2=0x7f3d7c1c3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 0x7f3d7c1c3360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c00fb00 comp rx=0 tx=0).stop 2024-09-16T11:03:57.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 msgr2=0x7f3d7c1c3360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.199+0000 7f3d78ff9700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1c2f80 0x7f3d7c1c3360 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c00fb00 comp rx=0 tx=0).stop 2024-09-16T11:03:57.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d8103e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:03:57.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d8103e700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 0x7f3d7c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d8103e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c09c470 con 0x7f3d7c1a0250 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 0x7f3d7c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient: get_auth_request con 0x7f3d7c1a0250 auth_method 0 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 10 monclient(hunting): handle_auth_done global_id 94144 payload 824 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 1 -- 172.21.15.5:0/1893789647 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d7c0a0680 con 0x7f3d7c1a0250 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d7b7fe700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 0x7f3d7c1a0660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 10 client.84278.objecter ms_handle_connect 0x7f3d7c1a0250 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 10 client.84278.objecter resend_mon_ops 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c022070 con 0x7f3d7c1a0250 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.211+0000 7f3d78ff9700 10 monclient: dump: 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.240 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f3d6c01c300 con 0x7f3d7c1a0250 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 msgr2=0x7f3d7c1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 0x7f3d7c1a0660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c001150 comp rx=0 tx=0).stop 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 msgr2=0x7f3d7c1a0660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d78ff9700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7c1a0250 0x7f3d7c1a0660 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c1a6390 tx=0x7f3d6c001150 comp rx=0 tx=0).stop 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 54, 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:03:57.242 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 2, 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.243 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.244 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:03:57.245 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:03:57.246 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.247 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:03:57.248 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:03:57.249 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.250 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:03:57.251 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.252 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.253 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:03:57.254 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 10 librados: watch_flush enter 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 10 librados: watch_flush exit 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 20 client.84278.objecter shutdown clearing up homeless session... 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 10 client.84278.objecter successfully canceled tick 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d68036f60 msgr2=0x7f3d68039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 --2- 172.21.15.5:0/1893789647 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d68036f60 0x7f3d68039420 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c0a1950 tx=0x7f3d74002030 comp rx=0 tx=0).stop 2024-09-16T11:03:57.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 10 monclient: shutdown 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 msgr2=0x7f3d7c1c5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 --2- 172.21.15.5:0/1893789647 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d7c071870 0x7f3d7c1c5f80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c075050 tx=0x7f3d70004770 comp rx=0 tx=0).stop 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8083d700 1 -- 172.21.15.5:0/1893789647 reap_dead start 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8083d700 1 -- 172.21.15.5:0/1893789647 reap_dead start 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 shutdown_connections 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 >> 172.21.15.5:0/1893789647 conn(0x7f3d7c0a45d0 msgr2=0x7f3d7c06ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 shutdown_connections 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 -- 172.21.15.5:0/1893789647 wait complete. 2024-09-16T11:03:57.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.215+0000 7f3d8103e700 1 librados: shutdown 2024-09-16T11:03:57.257 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:03:57.258 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T11:03:57.258 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T11:03:57.258 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T11:03:57.258 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-16T11:03:57.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-16T11:03:57.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 10 monclient: get_monmap_and_config 2024-09-16T11:03:57.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 10 monclient: monmap: 2024-09-16T11:03:57.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:57.379246+0000 2024-09-16T11:03:57.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:57.379246+0000 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:57.382 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:57.383 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:57.383 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 1 Processor -- start 2024-09-16T11:03:57.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 1 -- start start 2024-09-16T11:03:57.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 10 monclient: init 2024-09-16T11:03:57.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.375+0000 7f1dbd2b5700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T11:03:57.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1db8099e60 addr v1:172.21.15.143:6789/0 2024-09-16T11:03:57.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-b con 0x7f1db809f1e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-a con 0x7f1db809a950 addr v2:172.21.15.5:3300/0 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db809e3c0 con 0x7f1db8099e60 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db809e110 con 0x7f1db809a950 2024-09-16T11:03:57.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db80a0420 con 0x7f1db809f1e0 2024-09-16T11:03:57.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient(hunting): authenticate will time out at 1695.956218s 2024-09-16T11:03:57.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:47786/0 (socket says 172.21.15.5:47786) 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57974/0 (socket says 172.21.15.5:57974) 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 -- 172.21.15.5:0/624228465 learned_addr learned my addr 172.21.15.5:0/624228465 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): get_auth_request con 0x7f1db809f1e0 auth_method 0 2024-09-16T11:03:57.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): get_auth_request con 0x7f1db809a950 auth_method 0 2024-09-16T11:03:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 1 -- 172.21.15.5:0/624228465 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181427674 0 0) 0x7f1db809e3c0 con 0x7f1db8099e60 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient(hunting): handle_auth_done global_id 84281 payload 274 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 -- 172.21.15.5:0/624228465 >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 msgr2=0x7f1db813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:03:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 --2- 172.21.15.5:0/624228465 >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 -- 172.21.15.5:0/624228465 >> v1:172.21.15.143:6789/0 conn(0x7f1db8099e60 legacy=0x7f1db809a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 monclient: found mon.noname-b 2024-09-16T11:03:57.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:03:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 -- 172.21.15.5:0/624228465 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db809eae0 con 0x7f1db809f1e0 2024-09-16T11:03:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbcab4700 1 --2- 172.21.15.5:0/624228465 >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:03:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:03:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db77fe700 1 --2- 172.21.15.5:0/624228465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009910 tx=0x7f1da0009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0565c920a5829dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 1 -- 172.21.15.5:0/624228465 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011830 con 0x7f1db809f1e0 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: dump: 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.416 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 1 -- 172.21.15.5:0/624228465 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0011990 con 0x7f1db809f1e0 2024-09-16T11:03:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 5 monclient: authenticate success, global_id 84281 2024-09-16T11:03:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:03:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 1 -- 172.21.15.5:0/624228465 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011d40 con 0x7f1db809f1e0 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 4 set_mon_vals no callback set 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1db6ffd700 10 monclient: dump: 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.418 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.419 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.420 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.420 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.420 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 10 monclient: shutdown 2024-09-16T11:03:57.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/624228465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 msgr2=0x7f1db8138cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/624228465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009910 tx=0x7f1da0009c20 comp rx=0 tx=0).stop 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/624228465 shutdown_connections 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/624228465 >> v2:172.21.15.5:3300/0 conn(0x7f1db809a950 0x7f1db813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/624228465 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809f1e0 0x7f1db8138cd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/624228465 >> 172.21.15.5:0/624228465 conn(0x7f1db80a4450 msgr2=0x7f1db80a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/624228465 shutdown_connections 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.379+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/624228465 wait complete. 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: monmap: 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.422 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:57.423 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:57.423 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 librados: starting msgr at 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 librados: starting objecter 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 Processor -- start 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- start start 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 librados: setting wanted keys 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 librados: calling monclient init 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: init 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:03:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 0x7f1db81ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-a con 0x7f1db8099e60 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809a950 0x7f1db8133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-c con 0x7f1db809a950 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1db809f1e0 0x7f1db8134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): picked mon.noname-b con 0x7f1db809f1e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8122e90 con 0x7f1db8099e60 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db81ca8c0 con 0x7f1db809f1e0 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db81cb2f0 con 0x7f1db809a950 2024-09-16T11:03:57.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient(hunting): authenticate will time out at 1695.962788s 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 0x7f1db81ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 0x7f1db81ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57986/0 (socket says 172.21.15.5:57986) 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 -- 172.21.15.5:0/4098312294 learned_addr learned my addr 172.21.15.5:0/4098312294 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): get_auth_request con 0x7f1db8099e60 auth_method 0 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db77fe700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809a950 0x7f1db8133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db77fe700 10 monclient(hunting): get_auth_request con 0x7f1db809a950 auth_method 0 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient(hunting): handle_auth_done global_id 74463 payload 824 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809a950 msgr2=0x7f1db8133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809a950 0x7f1db8133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1db809f1e0 msgr2=0x7f1db8134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1db809f1e0 0x7f1db8134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 monclient: found mon.noname-a 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0009a70 con 0x7f1db8099e60 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db7fff700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 0x7f1db81ca380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1da80011e0 tx=0x7f1da800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 client.?.objecter ms_handle_connect 0x7f1db8099e60 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80054e0 con 0x7f1db8099e60 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: dump: 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 5 monclient: authenticate success, global_id 74463 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _renew_subs 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da80036a0 con 0x7f1db8099e60 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da0009570 con 0x7f1db8099e60 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8003a50 con 0x7f1db8099e60 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 client.74463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db67fc700 4 set_mon_vals no callback set 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1db4ff9700 10 monclient: dump: 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.430 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _renew_subs 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db809d0c0 con 0x7f1db8099e60 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.383+0000 7f1dbd2b5700 1 librados: init done 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1da8003bb0 con 0x7f1db8099e60 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbd2b5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 0x7f1db809b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8067380 con 0x7f1db809b560 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 --2- 172.21.15.5:0/4098312294 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1d98036f60 0x7f1d98039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 0x7f1db809b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient: get_auth_request con 0x7f1db809b560 auth_method 0 2024-09-16T11:03:57.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(575..575 src has 395..575) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f1da80474a0 con 0x7f1db8099e60 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 client.74463.objecter ms_dispatch 0x7f1db8000fd0 osd_map(575..575 src has 395..575) v4 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 3 client.74463.objecter handle_osd_map got epochs [575,575] > 0 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 3 client.74463.objecter handle_osd_map decoding full epoch 575 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 20 client.74463.objecter dump_active .. 0 homeless 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 10 monclient(hunting): handle_auth_done global_id 84287 payload 824 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbcab4700 1 --2- 172.21.15.5:0/4098312294 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1d98036f60 0x7f1d98039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1db809fb80 con 0x7f1db809b560 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbcab4700 10 monclient: get_auth_request con 0x7f1d98036f60 auth_method 0 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 client.74463.objecter ms_handle_connect 0x7f1db809b560 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 client.74463.objecter resend_mon_ops 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db77fe700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 0x7f1db809b940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009910 tx=0x7f1da0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da001ca60 con 0x7f1db809b560 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 monclient: dump: 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 client.74463.objecter ms_handle_connect 0x7f1d98036f60 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1dbcab4700 1 --2- 172.21.15.5:0/4098312294 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1d98036f60 0x7f1d98039420 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e330 tx=0x7f1db000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f1da001cbc0 con 0x7f1db809b560 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 msgr2=0x7f1db809b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 0x7f1db809b940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009910 tx=0x7f1da0004770 comp rx=0 tx=0).stop 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 msgr2=0x7f1db809b940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.387+0000 7f1db4ff9700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809b560 0x7f1db809b940 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1dbd2b5700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T11:03:57.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 0x7f1db80a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1dbd2b5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db806caf0 con 0x7f1db809be80 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 0x7f1db80a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient: get_auth_request con 0x7f1db809be80 auth_method 0 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.399+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db77fe700 10 monclient(hunting): handle_auth_done global_id 84290 payload 824 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db77fe700 1 -- 172.21.15.5:0/4098312294 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f1db80a0140 con 0x7f1db809be80 2024-09-16T11:03:57.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 client.74463.objecter ms_handle_connect 0x7f1db809be80 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 client.74463.objecter resend_mon_ops 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db77fe700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 0x7f1db80a1f10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000c00 tx=0x7f1da0011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da001a810 con 0x7f1db809be80 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 monclient: dump: 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.438 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (secure 0 0 0) 0x7f1da001a970 con 0x7f1db809be80 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 msgr2=0x7f1db80a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 0x7f1db80a1f10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000c00 tx=0x7f1da0011000 comp rx=0 tx=0).stop 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 msgr2=0x7f1db80a1f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1db4ff9700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1db809be80 0x7f1db80a1f10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.439 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:03:57.440 INFO:teuthology.orchestra.run.smithi005.stdout: "ret": 0, 2024-09-16T11:03:57.440 INFO:teuthology.orchestra.run.smithi005.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-16T11:03:57.440 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:03:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 10 librados: watch_flush enter 2024-09-16T11:03:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 10 librados: watch_flush exit 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 20 client.74463.objecter shutdown clearing up homeless session... 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 10 client.74463.objecter successfully canceled tick 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1d98036f60 msgr2=0x7f1d98039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/4098312294 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1d98036f60 0x7f1d98039420 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e330 tx=0x7f1db000b040 comp rx=0 tx=0).stop 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 10 monclient: shutdown 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 msgr2=0x7f1db81ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 --2- 172.21.15.5:0/4098312294 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1db8099e60 0x7f1db81ca380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1da80011e0 tx=0x7f1da800cae0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbcab4700 1 -- 172.21.15.5:0/4098312294 reap_dead start 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbcab4700 1 -- 172.21.15.5:0/4098312294 reap_dead start 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 shutdown_connections 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 >> 172.21.15.5:0/4098312294 conn(0x7f1db80a4450 msgr2=0x7f1db806dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 shutdown_connections 2024-09-16T11:03:57.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 -- 172.21.15.5:0/4098312294 wait complete. 2024-09-16T11:03:57.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.403+0000 7f1dbd2b5700 1 librados: shutdown 2024-09-16T11:03:57.443 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T11:03:57.443 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T11:03:57.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:03:57.491 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T11:03:57.491 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T11:03:57.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: get_monmap_and_config 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: monmap: 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.607 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:03:57.606142+0000 2024-09-16T11:03:57.608 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:03:57.606142+0000 2024-09-16T11:03:57.608 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.608 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.608 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:03:57.608 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:03:57.609 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:03:57.609 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:03:57.609 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:03:57.609 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 Processor -- start 2024-09-16T11:03:57.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 -- start start 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: init 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c0a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-b con 0x7ff88c0a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff88c0a4280 0x7ff88c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-c con 0x7ff88c0a4280 addr v2:172.21.15.143:3300/0 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff88c0a3d10 addr v1:172.21.15.5:6789/0 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c09fdc0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c0a1e50 con 0x7ff88c0a4280 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c09fae0 con 0x7ff88c0a2eb0 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff892ad1700 10 monclient(hunting): authenticate will time out at 1696.182645s 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff89086d700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 msgr2=0x7ff88c0a3290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff89086d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c0a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff89106e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff88c0a3d10 0x7ff88c135c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49544/0 (socket says 172.21.15.5:49544) 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff89106e700 1 -- 172.21.15.5:0/2376656969 learned_addr learned my addr 172.21.15.5:0/2376656969 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 --2- 172.21.15.5:0/2376656969 >> v2:172.21.15.143:3300/0 conn(0x7ff88c0a4280 0x7ff88c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): get_auth_request con 0x7ff88c0a4280 auth_method 0 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829171946 0 0) 0x7ff88c09fdc0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 10 monclient(hunting): _negotiate 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 10 monclient(hunting): my global_id is 74466 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8780035d0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient(hunting): handle_auth_done global_id 94150 payload 274 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 -- 172.21.15.5:0/2376656969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 msgr2=0x7ff88c0a3290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:03:57.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 --2- 172.21.15.5:0/2376656969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c0a3290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 -- 172.21.15.5:0/2376656969 >> v1:172.21.15.5:6789/0 conn(0x7ff88c0a3d10 legacy=0x7ff88c135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1640546898 0 0) 0x7ff8780035d0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 monclient: found mon.noname-c 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88bfff700 1 -- 172.21.15.5:0/2376656969 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c09efb0 con 0x7ff88c0a4280 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.603+0000 7ff88b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:03:57.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88bfff700 1 --2- 172.21.15.5:0/2376656969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c0a37d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff884009d90 tx=0x7ff88400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91351c2826c4fc0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88400e070 con 0x7ff88c0a4280 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: dump: 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.617 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 5 monclient: authenticate success, global_id 94150 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884003a40 con 0x7ff88c0a4280 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 10 monclient: get_monmap_and_config success 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 4 set_mon_vals no callback set 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 1 -- 172.21.15.5:0/2376656969 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874001670 con 0x7ff88c0a4280 2024-09-16T11:03:57.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff88b7fe700 10 monclient: dump: 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 10 monclient: shutdown 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 -- 172.21.15.5:0/2376656969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 msgr2=0x7ff88c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 --2- 172.21.15.5:0/2376656969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c0a37d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff884009d90 tx=0x7ff88400b330 comp rx=0 tx=0).stop 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 -- 172.21.15.5:0/2376656969 shutdown_connections 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 --2- 172.21.15.5:0/2376656969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c0a37d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 --2- 172.21.15.5:0/2376656969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c0a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 -- 172.21.15.5:0/2376656969 >> 172.21.15.5:0/2376656969 conn(0x7ff88c0a4fb0 msgr2=0x7ff88c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 -- 172.21.15.5:0/2376656969 shutdown_connections 2024-09-16T11:03:57.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 1 -- 172.21.15.5:0/2376656969 wait complete. 2024-09-16T11:03:57.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 10 monclient: build_initial_monmap 2024-09-16T11:03:57.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.607+0000 7ff892ad1700 10 monclient: monmap: 2024-09-16T11:03:57.625 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:03:57.625 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:03:57.625 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:03:57.627 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 librados: starting msgr at 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 librados: starting objecter 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 Processor -- start 2024-09-16T11:03:57.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- start start 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 librados: setting wanted keys 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 librados: calling monclient init 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: init 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _reopen_session rank -1 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c1cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-c con 0x7ff88c0a2eb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-a con 0x7ff88c0a3d10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c122490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff89086d700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 msgr2=0x7ff88c1cab30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): picked mon.noname-b con 0x7ff88c0a4280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c126410 con 0x7ff88c0a3d10 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c09f6a0 con 0x7ff88c0a4280 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): start opening mon connection 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0a7570 con 0x7ff88c0a2eb0 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff89086d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c1cab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): _renew_subs 2024-09-16T11:03:57.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient(hunting): authenticate will time out at 1696.189982s 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58010/0 (socket says 172.21.15.5:58010) 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 -- 172.21.15.5:0/3310346332 learned_addr learned my addr 172.21.15.5:0/3310346332 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): get_auth_request con 0x7ff88c0a3d10 auth_method 0 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:03:57.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient(hunting): handle_auth_done global_id 74469 payload 824 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff89106e700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c122490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 -- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 msgr2=0x7ff88c1cab30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c1cab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 -- 172.21.15.5:0/3310346332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 msgr2=0x7ff88c122490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c122490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 monclient: found mon.noname-a 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 -- 172.21.15.5:0/3310346332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884009a70 con 0x7ff88c0a3d10 2024-09-16T11:03:57.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff89106e700 10 monclient: get_auth_request con 0x7ff88c0a4280 auth_method 0 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff89106e700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c122490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88bfff700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06c4d0 tx=0x7ff884004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 client.?.objecter ms_handle_connect 0x7ff88c0a3d10 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884004720 con 0x7ff88c0a3d10 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: dump: 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.633 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: _finish_auth 0 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: _check_auth_tickets 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 5 monclient: authenticate success, global_id 74469 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _renew_subs 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884004880 con 0x7ff88c0a3d10 2024-09-16T11:03:57.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c0a7040 con 0x7ff88c0a3d10 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88401e670 con 0x7ff88c0a3d10 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff88affd700 4 set_mon_vals no callback set 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 client.74469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff8897fa700 10 monclient: dump: 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:03:57.635 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _renew_subs 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c126130 con 0x7ff88c0a3d10 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.611+0000 7ff892ad1700 1 librados: init done 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff88401e7d0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 1 --2- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 0x7ff86c039400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(575..575 src has 395..575) v4 ==== 5747+0+0 (secure 0 0 0) 0x7ff8840496f0 con 0x7ff88c0a3d10 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 10 client.74469.objecter ms_dispatch 0x7ff88c000fd0 osd_map(575..575 src has 395..575) v4 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 3 client.74469.objecter handle_osd_map got epochs [575,575] > 0 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 3 client.74469.objecter handle_osd_map decoding full epoch 575 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 20 client.74469.objecter dump_active .. 0 homeless 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff892ad1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff892ad1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff892ad1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c125c00 con 0x7ff88c0a3d10 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff89086d700 1 --2- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 0x7ff86c039400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:03:57.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff89086d700 10 monclient: get_auth_request con 0x7ff86c036f40 auth_method 0 2024-09-16T11:03:57.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff8897fa700 10 client.74469.objecter ms_handle_connect 0x7ff86c036f40 2024-09-16T11:03:57.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.615+0000 7ff89086d700 1 --2- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 0x7ff86c039400 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c006de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:03:57.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.623+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff884024900 con 0x7ff88c0a3d10 2024-09-16T11:03:57.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.623+0000 7ff8897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:03:57.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.623+0000 7ff8897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:03:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff892ad1700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:03:57.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff892ad1700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff892ad1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:03:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff88c136760 con 0x7ff88c0a3d10 2024-09-16T11:03:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff8897fa700 1 -- 172.21.15.5:0/3310346332 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ff884017020 con 0x7ff88c0a3d10 2024-09-16T11:03:57.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff8897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:03:57.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.859+0000 7ff8897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:03:57.865 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:03:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 10 librados: watch_flush enter 2024-09-16T11:03:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 10 librados: watch_flush exit 2024-09-16T11:03:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 20 client.74469.objecter shutdown clearing up homeless session... 2024-09-16T11:03:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 10 client.74469.objecter successfully canceled tick 2024-09-16T11:03:57.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 msgr2=0x7ff86c039400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 0x7ff86c039400 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c006de0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 10 monclient: shutdown 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 msgr2=0x7ff88c1cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06c4d0 tx=0x7ff884004af0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 shutdown_connections 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff88c0a4280 0x7ff88c122490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff86c036f40 0x7ff86c039400 unknown :-1 s=CLOSED pgs=3874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff88c0a3d10 0x7ff88c1cb070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 --2- 172.21.15.5:0/3310346332 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff88c0a2eb0 0x7ff88c1cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:03:57.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 >> 172.21.15.5:0/3310346332 conn(0x7ff88c0a4fb0 msgr2=0x7ff88c09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:03:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 shutdown_connections 2024-09-16T11:03:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 -- 172.21.15.5:0/3310346332 wait complete. 2024-09-16T11:03:57.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:03:57.863+0000 7ff892ad1700 1 librados: shutdown 2024-09-16T11:04:00.880 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.995+0000 7f798cafc700 10 monclient: get_monmap_and_config 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.995+0000 7f798cafc700 10 monclient: build_initial_monmap 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.995+0000 7f798cafc700 10 monclient: monmap: 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:04:00.999717+0000 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:04:00.999717+0000 2024-09-16T11:04:01.001 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:04:01.002 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 Processor -- start 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- start start 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient: init 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f79880a2bf0 0x7f79880a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-a con 0x7f79880a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f798809f780 0x7f79880a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-c con 0x7f798809f780 addr v2:172.21.15.143:3300/0 2024-09-16T11:04:01.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79881359b0 addr v1:172.21.15.143:6789/0 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79880a3a50 con 0x7f79881359b0 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79880a3c40 con 0x7f79880a2bf0 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79880a4300 con 0x7f798809f780 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient(hunting): authenticate will time out at 1699.576831s 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f79880a2bf0 0x7f79880a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f798809f780 0x7f79880a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f79880a2bf0 0x7f79880a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58020/0 (socket says 172.21.15.5:58020) 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 -- 172.21.15.5:0/782458298 learned_addr learned my addr 172.21.15.5:0/782458298 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): get_auth_request con 0x7f79880a2bf0 auth_method 0 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): get_auth_request con 0x7f798809f780 auth_method 0 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:01.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f797ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient(hunting): handle_auth_done global_id 74472 payload 274 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 -- 172.21.15.5:0/782458298 >> v2:172.21.15.143:3300/0 conn(0x7f798809f780 msgr2=0x7f79880a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 --2- 172.21.15.5:0/782458298 >> v2:172.21.15.143:3300/0 conn(0x7f798809f780 0x7f79880a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 -- 172.21.15.5:0/782458298 >> v1:172.21.15.143:6789/0 conn(0x7f79881359b0 legacy=0x7f7988135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient: _finish_hunting 0 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 monclient: found mon.noname-a 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:01.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 -- 172.21.15.5:0/782458298 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798809fda0 con 0x7f79880a2bf0 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f79877fe700 1 --2- 172.21.15.5:0/782458298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79880a2bf0 0x7f79880a2fd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79880a60a0 tx=0x7f7978009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9728090d2e67a64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 1 -- 172.21.15.5:0/782458298 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978011830 con 0x7f79880a2bf0 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: dump: 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.016 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: _finish_auth 0 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 1 -- 172.21.15.5:0/782458298 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978011990 con 0x7f79880a2bf0 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 5 monclient: authenticate success, global_id 74472 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:01.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 1 -- 172.21.15.5:0/782458298 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978011d20 con 0x7f79880a2bf0 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient: get_monmap_and_config success 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 4 set_mon_vals no callback set 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f7986ffd700 10 monclient: dump: 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:01.018 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 10 monclient: shutdown 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- 172.21.15.5:0/782458298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79880a2bf0 msgr2=0x7f79880a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 --2- 172.21.15.5:0/782458298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79880a2bf0 0x7f79880a2fd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f79880a60a0 tx=0x7f7978009bf0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- 172.21.15.5:0/782458298 shutdown_connections 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 --2- 172.21.15.5:0/782458298 >> v2:172.21.15.143:3300/0 conn(0x7f798809f780 0x7f79880a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 --2- 172.21.15.5:0/782458298 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79880a2bf0 0x7f79880a2fd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- 172.21.15.5:0/782458298 >> 172.21.15.5:0/782458298 conn(0x7f79880a53e0 msgr2=0x7f7988073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:01.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- 172.21.15.5:0/782458298 shutdown_connections 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:00.999+0000 7f798cafc700 1 -- 172.21.15.5:0/782458298 wait complete. 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: build_initial_monmap 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: monmap: 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:04:01.020 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 librados: starting msgr at 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 librados: starting objecter 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 Processor -- start 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- start start 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 librados: setting wanted keys 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 librados: calling monclient init 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: init 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:04:01.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-a con 0x7f798809f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 0x7f7988135190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-c con 0x7f79880a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79881359b0 0x7f798811e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): picked mon.noname-b con 0x7f79881359b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79880a66d0 con 0x7f798809f780 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79880704f0 con 0x7f79881359b0 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:01.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798809f4e0 con 0x7f79880a2bf0 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient(hunting): authenticate will time out at 1699.581605s 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f797ffff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 msgr2=0x7f7988135190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58022/0 (socket says 172.21.15.5:58022) 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f797ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 0x7f7988135190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 -- 172.21.15.5:0/185364758 learned_addr learned my addr 172.21.15.5:0/185364758 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): get_auth_request con 0x7f798809f780 auth_method 0 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:01.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7987fff700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79881359b0 0x7f798811e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7987fff700 10 monclient(hunting): get_auth_request con 0x7f79881359b0 auth_method 0 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7987fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7987fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7987fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient(hunting): handle_auth_done global_id 74475 payload 824 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 -- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 msgr2=0x7f7988135190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 0x7f7988135190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 -- 172.21.15.5:0/185364758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79881359b0 msgr2=0x7f798811e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79881359b0 0x7f798811e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient: _finish_hunting 0 2024-09-16T11:04:01.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 monclient: found mon.noname-a 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 -- 172.21.15.5:0/185364758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009570 con 0x7f798809f780 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79877fe700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7988074a80 tx=0x7f7978004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 client.?.objecter ms_handle_connect 0x7f798809f780 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978004340 con 0x7f798809f780 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: dump: 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:01.025 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: _finish_auth 0 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:01.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 5 monclient: authenticate success, global_id 74475 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797801f070 con 0x7f798809f780 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _renew_subs 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7988122330 con 0x7f798809f780 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780036c0 con 0x7f798809f780 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 10 monclient: dump: 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:01.027 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f79867fc700 4 set_mon_vals no callback set 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 client.74475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _renew_subs 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7988120c00 con 0x7f798809f780 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f798cafc700 1 librados: init done 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.003+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7978021030 con 0x7f798809f780 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(575..575 src has 395..575) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f79780491e0 con 0x7f798809f780 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 10 client.74475.objecter ms_dispatch 0x7f7988000fd0 osd_map(575..575 src has 395..575) v4 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 3 client.74475.objecter handle_osd_map got epochs [575,575] > 0 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 3 client.74475.objecter handle_osd_map decoding full epoch 575 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f797ffff700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f797ffff700 1 -- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 msgr2=0x7f7968039400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f797ffff700 1 -- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 msgr2=0x7f7968039400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f797ffff700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f797ffff700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:01.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f7984ff9700 20 client.74475.objecter dump_active .. 0 homeless 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f798cafc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f798cafc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f798cafc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.007+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79880a2fd0 con 0x7f798809f780 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.015+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7978015080 con 0x7f798809f780 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.015+0000 7f7984ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.015+0000 7f7984ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:04:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.211+0000 7f797ffff700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.211+0000 7f797ffff700 10 monclient: get_auth_request con 0x7f7968036f40 auth_method 0 2024-09-16T11:04:01.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.211+0000 7f7984ff9700 10 client.74475.objecter ms_handle_connect 0x7f7968036f40 2024-09-16T11:04:01.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.211+0000 7f797ffff700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f798811feb0 tx=0x7f7970008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:01.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:04:01.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:01.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:01.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7988121d70 con 0x7f798809f780 2024-09-16T11:04:01.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f7984ff9700 1 -- 172.21.15.5:0/185364758 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f797802c960 con 0x7f798809f780 2024-09-16T11:04:01.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f7984ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:01.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f7984ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:04:01.255 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:04:01.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 librados: watch_flush enter 2024-09-16T11:04:01.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 librados: watch_flush exit 2024-09-16T11:04:01.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 20 client.74475.objecter shutdown clearing up homeless session... 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 client.74475.objecter successfully canceled tick 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 msgr2=0x7f7968039400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f798811feb0 tx=0x7f7970008040 comp rx=0 tx=0).stop 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 10 monclient: shutdown 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 msgr2=0x7f7988134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7988074a80 tx=0x7f7978004770 comp rx=0 tx=0).stop 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 shutdown_connections 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79881359b0 0x7f798811e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f79880a2bf0 0x7f7988135190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7968036f40 0x7f7968039400 unknown :-1 s=CLOSED pgs=3875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 --2- 172.21.15.5:0/185364758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f798809f780 0x7f7988134c50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:01.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 >> 172.21.15.5:0/185364758 conn(0x7f79880a53e0 msgr2=0x7f7988136140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:01.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 shutdown_connections 2024-09-16T11:04:01.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 -- 172.21.15.5:0/185364758 wait complete. 2024-09-16T11:04:01.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:01.251+0000 7f798cafc700 1 librados: shutdown 2024-09-16T11:04:04.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:04:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: get_monmap_and_config 2024-09-16T11:04:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: build_initial_monmap 2024-09-16T11:04:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: monmap: 2024-09-16T11:04:04.391 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:04:04.390133+0000 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:04:04.390133+0000 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:04:04.392 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:04:04.393 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 Processor -- start 2024-09-16T11:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 -- start start 2024-09-16T11:04:04.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: init 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec7c09b3e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec7c09bed0 0x7fec7c1378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-c con 0x7fec7c09bed0 addr v2:172.21.15.143:3300/0 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 0x7fec7c139e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-b con 0x7fec7c09c2e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7c09fd30 con 0x7fec7c09b3e0 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7c09e350 con 0x7fec7c09bed0 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:04.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c09ca20 con 0x7fec7c09c2e0 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81d9a700 10 monclient(hunting): authenticate will time out at 1702.966675s 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81599700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 msgr2=0x7fec7c139e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec81599700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 0x7fec7c139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec7c09bed0 0x7fec7c1378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec7c09bed0 0x7fec7c1378f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:57972/0 (socket says 172.21.15.5:57972) 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 -- 172.21.15.5:0/2314911560 learned_addr learned my addr 172.21.15.5:0/2314911560 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): get_auth_request con 0x7fec7c09bed0 auth_method 0 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:04.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient(hunting): handle_auth_done global_id 94159 payload 274 2024-09-16T11:04:04.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 -- 172.21.15.5:0/2314911560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 msgr2=0x7fec7c139e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 --2- 172.21.15.5:0/2314911560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 0x7fec7c139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 -- 172.21.15.5:0/2314911560 >> v1:172.21.15.143:6789/0 conn(0x7fec7c09b3e0 legacy=0x7fec7c09b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 monclient: found mon.noname-c 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.387+0000 7fec7bfff700 1 -- 172.21.15.5:0/2314911560 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c09cd00 con 0x7fec7c09bed0 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7bfff700 1 --2- 172.21.15.5:0/2314911560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09bed0 0x7fec7c1378f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec6400bbe0 tx=0x7fec6400bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23061b023f967bcd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 1 -- 172.21.15.5:0/2314911560 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6400dda0 con 0x7fec7c09bed0 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:04.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: dump: 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 1 -- 172.21.15.5:0/2314911560 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec64010470 con 0x7fec7c09bed0 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 1 -- 172.21.15.5:0/2314911560 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64010800 con 0x7fec7c09bed0 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 5 monclient: authenticate success, global_id 94159 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 10 monclient: get_monmap_and_config success 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 4 set_mon_vals no callback set 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec7b7fe700 10 monclient: dump: 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:04.402 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 10 monclient: shutdown 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:04.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 -- 172.21.15.5:0/2314911560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09bed0 msgr2=0x7fec7c1378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 --2- 172.21.15.5:0/2314911560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09bed0 0x7fec7c1378f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec6400bbe0 tx=0x7fec6400bef0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 -- 172.21.15.5:0/2314911560 shutdown_connections 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 --2- 172.21.15.5:0/2314911560 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09c2e0 0x7fec7c139e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 --2- 172.21.15.5:0/2314911560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09bed0 0x7fec7c1378f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 -- 172.21.15.5:0/2314911560 >> 172.21.15.5:0/2314911560 conn(0x7fec7c0a45d0 msgr2=0x7fec7c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 -- 172.21.15.5:0/2314911560 shutdown_connections 2024-09-16T11:04:04.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 -- 172.21.15.5:0/2314911560 wait complete. 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 10 monclient: build_initial_monmap 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 10 monclient: monmap: 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:04:04.412 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 librados: starting msgr at 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.391+0000 7fec81d9a700 1 librados: starting objecter 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 Processor -- start 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- start start 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 librados: setting wanted keys 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 librados: calling monclient init 2024-09-16T11:04:04.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: init 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09b3e0 0x7fec7c1c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-b con 0x7fec7c09b3e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 0x7fec7c1c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-c con 0x7fec7c09bed0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): picked mon.noname-a con 0x7fec7c09c2e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1a73c0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:04.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c0a35b0 con 0x7fec7c09b3e0 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c070fc0 con 0x7fec7c09bed0 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient(hunting): authenticate will time out at 1702.973406s 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec7bfff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 msgr2=0x7fec7c1c0300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec7bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 0x7fec7c1c0300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58326/0 (socket says 172.21.15.5:58326) 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 -- 172.21.15.5:0/1316783422 learned_addr learned my addr 172.21.15.5:0/1316783422 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09b3e0 0x7fec7c1c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:04.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): get_auth_request con 0x7fec7c09c2e0 auth_method 0 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): get_auth_request con 0x7fec7c09b3e0 auth_method 0 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:04.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec80d98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient(hunting): handle_auth_done global_id 74478 payload 824 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 -- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 msgr2=0x7fec7c1c0300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 0x7fec7c1c0300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 -- 172.21.15.5:0/1316783422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09b3e0 msgr2=0x7fec7c1c2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09b3e0 0x7fec7c1c2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient: _finish_hunting 0 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 monclient: found mon.noname-a 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 -- 172.21.15.5:0/1316783422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6400b810 con 0x7fec7c09c2e0 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81599700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec7400be80 tx=0x7fec74009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:04.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 client.?.objecter ms_handle_connect 0x7fec7c09c2e0 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7400ccb0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: dump: 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:04.418 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: _finish_auth 0 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: _check_auth_tickets 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 5 monclient: authenticate success, global_id 74478 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7400ce10 con 0x7fec7c09c2e0 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _renew_subs 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:04.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c09b0d0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74016670 con 0x7fec7c09c2e0 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec797fa700 10 monclient: dump: 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:04.420 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec7a7fc700 4 set_mon_vals no callback set 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 client.74478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _renew_subs 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c1a7c60 con 0x7fec7c09c2e0 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.395+0000 7fec81d9a700 1 librados: init done 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fec74012020 con 0x7fec7c09c2e0 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec81d9a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec81d9a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:04.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec81d9a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c09b7c0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 1 --2- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 0x7fec600393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(575..575 src has 395..575) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fec74048290 con 0x7fec7c09c2e0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 10 client.74478.objecter ms_dispatch 0x7fec7c000fd0 osd_map(575..575 src has 395..575) v4 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 3 client.74478.objecter handle_osd_map got epochs [575,575] > 0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 3 client.74478.objecter handle_osd_map decoding full epoch 575 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 20 client.74478.objecter dump_active .. 0 homeless 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec80d98700 1 --2- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 0x7fec600393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec80d98700 10 monclient: get_auth_request con 0x7fec60036f10 auth_method 0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec797fa700 10 client.74478.objecter ms_handle_connect 0x7fec60036f10 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.399+0000 7fec80d98700 1 --2- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 0x7fec600393d0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0018a0 tx=0x7fec6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:04.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.407+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fec740076a0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.407+0000 7fec797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:04.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.407+0000 7fec797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.639+0000 7fec81d9a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.639+0000 7fec81d9a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.639+0000 7fec81d9a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.639+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fec7c09e010 con 0x7fec7c09c2e0 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec797fa700 1 -- 172.21.15.5:0/1316783422 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1536 (secure 0 0 0) 0x7fec740072e0 con 0x7fec7c09c2e0 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:04.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:04:04.647 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 10 librados: watch_flush enter 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 10 librados: watch_flush exit 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 20 client.74478.objecter shutdown clearing up homeless session... 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 10 client.74478.objecter successfully canceled tick 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 msgr2=0x7fec600393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 0x7fec600393d0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0018a0 tx=0x7fec6c009040 comp rx=0 tx=0).stop 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 10 monclient: shutdown 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 msgr2=0x7fec7c1c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec7400be80 tx=0x7fec74009f40 comp rx=0 tx=0).stop 2024-09-16T11:04:04.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 shutdown_connections 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> v2:172.21.15.143:6812/1098205034 conn(0x7fec60036f10 0x7fec600393d0 unknown :-1 s=CLOSED pgs=3876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fec7c09c2e0 0x7fec7c1c62b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fec7c09bed0 0x7fec7c1c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 --2- 172.21.15.5:0/1316783422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec7c09b3e0 0x7fec7c1c2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 >> 172.21.15.5:0/1316783422 conn(0x7fec7c0a45d0 msgr2=0x7fec7c06f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 shutdown_connections 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 -- 172.21.15.5:0/1316783422 wait complete. 2024-09-16T11:04:04.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:04.643+0000 7fec81d9a700 1 librados: shutdown 2024-09-16T11:04:07.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:04:07.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: get_monmap_and_config 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: build_initial_monmap 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: monmap: 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:04:07.781590+0000 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:04:07.781590+0000 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:04:07.783 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:04:07.784 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 Processor -- start 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 -- start start 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: init 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb55c09e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-b con 0x7fb55c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:07.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-a con 0x7fb55c09dfd0 addr v2:172.21.15.5:3300/0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c070e40 con 0x7fb55c09e8e0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55c0a01d0 con 0x7fb55c09dfd0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c09fef0 con 0x7fb55c09d6a0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561e0a700 10 monclient(hunting): authenticate will time out at 1706.358628s 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb55bfff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 msgr2=0x7fb55c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb55bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb560e08700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb55c09e8e0 0x7fb55c09ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34386/0 (socket says 172.21.15.5:34386) 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 learned_addr learned my addr 172.21.15.5:0/365127799 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:04:07.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb55c09dfd0 auth_method 0 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:07.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.779+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb55bfff700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 msgr2=0x7fb55c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb55bfff700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb55c09dfd0 auth_method 0 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:07.983+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb55bfff700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 msgr2=0x7fb55c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb55bfff700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb55c09dfd0 auth_method 0 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:08.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:08.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:04:08.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:08.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:08.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:08.387+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb55bfff700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 msgr2=0x7fb55c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb55bfff700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb55c09dfd0 auth_method 0 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:09.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:09.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:09.191+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): tick 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 msgr2=0x7fb55c138b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:10.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb55c09d6a0 0x7fb55c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 msgr2=0x7fb55c13b0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.5:3300/0 conn(0x7fb55c09dfd0 0x7fb55c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 -- 172.21.15.5:0/365127799 >> v1:172.21.15.5:6789/0 conn(0x7fb55c09e8e0 legacy=0x7fb55c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient: _add_conns ranks=[4,1] 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fb540003760 addr v2:172.21.15.143:3300/0 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5400062b0 addr v1:172.21.15.143:6789/0 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 -- 172.21.15.5:0/365127799 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c09f9f0 con 0x7fb5400062b0 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 1 -- 172.21.15.5:0/365127799 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55c0a01d0 con 0x7fb540003760 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb55a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:10.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 10 monclient(hunting): get_auth_request con 0x7fb540003760 auth_method 0 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:10.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.783+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 10 monclient(hunting): get_auth_request con 0x7fb540003760 auth_method 0 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:10.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:10.987+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 10 monclient(hunting): get_auth_request con 0x7fb540003760 auth_method 0 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:11.391+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 10 monclient(hunting): get_auth_request con 0x7fb540003760 auth_method 0 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:04:12.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.195+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:04:12.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033071041 0 0) 0x7fb55c09f9f0 con 0x7fb5400062b0 2024-09-16T11:04:12.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): _negotiate 2024-09-16T11:04:12.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): my global_id is 94165 2024-09-16T11:04:12.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c0035d0 con 0x7fb5400062b0 2024-09-16T11:04:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124438696 0 0) 0x7fb54c0035d0 con 0x7fb5400062b0 2024-09-16T11:04:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 msgr2=0x7fb540005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 --2- 172.21.15.5:0/365127799 >> v2:172.21.15.143:3300/0 conn(0x7fb540003760 0x7fb540005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 monclient: found mon.noname-c-legacy 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54000b040 con 0x7fb5400062b0 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7fb5480043f0 con 0x7fb5400062b0 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb561e0a700 5 monclient: authenticate success, global_id 94165 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb561e0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:12.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: dump: 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 >> v1:172.21.15.143:6789/0 conn(0x7fb5400062b0 legacy=0x7fb540008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 reap_dead start 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 0x7fb54c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): picked mon.b con 0x7fb54c003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 0x7fb54c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 10 monclient(hunting): picked mon.c con 0x7fb54c004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.283+0000 7fb55b7fe700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb54c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): picked mon.a con 0x7fb54c005280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c0a01d0 con 0x7fb54c005280 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c1b0 con 0x7fb54c003a20 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c370 con 0x7fb54c004870 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55bfff700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 msgr2=0x7fb54c004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561e0a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb55bfff700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 0x7fb54c004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb54c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 0x7fb54c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb54c005280 auth_method 0 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): get_auth_request con 0x7fb54c003a20 auth_method 0 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient(hunting): handle_auth_done global_id 94165 payload 293 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 msgr2=0x7fb54c004cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 0x7fb54c004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 msgr2=0x7fb54c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 0x7fb54c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 monclient: found mon.a 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 -- 172.21.15.5:0/365127799 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0035d0 con 0x7fb54c005280 2024-09-16T11:04:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb560e08700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 0x7fb54c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:04:12.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.287+0000 7fb561609700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb54c00ba30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0a1820 tx=0x7fb54c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85be435505764a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.312 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.312 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8205 ms) 2024-09-16T11:04:12.312 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5962 ms) 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5044 ms) 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (24970 ms) 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13558 ms) 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T11:04:12.313 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11477 ms) 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 00000012 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12040 ms) 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: require_osd_release = reef 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5960 ms) 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (616420 ms total) 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1076 ms) 2024-09-16T11:04:12.314 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10935 ms) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (31951 ms) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (43965 ms total) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6990 ms) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8055 ms) 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T11:04:12.315 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: waiting for scrub... 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: done waiting 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24505 ms) 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9479 ms) 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8056 ms) 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7904 ms) 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10066 ms) 2024-09-16T11:04:12.316 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.307+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554004160 con 0x7fb54c005280 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.307+0000 7fb55b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.307+0000 7fb55b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.307+0000 7fb55b7fe700 10 monclient: dump: 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.317 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.307+0000 7fb561e0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb554004300 con 0x7fb54c005280 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 1 -- 172.21.15.5:0/365127799 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5540046b0 con 0x7fb54c005280 2024-09-16T11:04:12.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb55b7fe700 10 monclient: dump: 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb561e0a700 10 monclient: get_monmap_and_config success 2024-09-16T11:04:12.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.311+0000 7fb561e0a700 4 set_mon_vals no callback set 2024-09-16T11:04:12.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 10 monclient: shutdown 2024-09-16T11:04:12.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 -- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 msgr2=0x7fb54c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb54c00ba30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0a1820 tx=0x7fb54c0058a0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 -- 172.21.15.5:0/365127799 shutdown_connections 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb54c00ba30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c004870 0x7fb54c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 --2- 172.21.15.5:0/365127799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c003a20 0x7fb54c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 -- 172.21.15.5:0/365127799 >> 172.21.15.5:0/365127799 conn(0x7fb55c0a45d0 msgr2=0x7fb55c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 -- 172.21.15.5:0/365127799 shutdown_connections 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 -- 172.21.15.5:0/365127799 wait complete. 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 10 monclient: build_initial_monmap 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 10 monclient: monmap: 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:04:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 librados: starting msgr at 2024-09-16T11:04:12.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.315+0000 7fb561e0a700 1 librados: starting objecter 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 Processor -- start 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 -- start start 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 librados: setting wanted keys 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 librados: calling monclient init 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient: init 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 0x7fb55c1c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-c con 0x7fb54c003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 0x7fb55c133bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-b con 0x7fb54c004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb55c134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): picked mon.noname-a con 0x7fb54c005280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb560e08700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 msgr2=0x7fb55c1c7e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c122a40 con 0x7fb54c005280 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c1c8bb0 con 0x7fb54c004870 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c123dd0 con 0x7fb54c003a20 2024-09-16T11:04:12.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb560e08700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 0x7fb55c1c7e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561e0a700 10 monclient(hunting): authenticate will time out at 1710.897668s 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb55c134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb55c134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58378/0 (socket says 172.21.15.5:58378) 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 0x7fb55c133bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 1 -- 172.21.15.5:0/3542714751 learned_addr learned my addr 172.21.15.5:0/3542714751 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): get_auth_request con 0x7fb54c005280 auth_method 0 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): get_auth_request con 0x7fb54c004870 auth_method 0 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb561609700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient(hunting): handle_auth_done global_id 94168 payload 824 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 -- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 msgr2=0x7fb55c1c7e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 0x7fb55c1c7e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 -- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 msgr2=0x7fb55c134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb55c134130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 monclient: found mon.noname-b 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 -- 172.21.15.5:0/3542714751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c1346a0 con 0x7fb54c004870 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb5597fa700 10 client.?.objecter ms_handle_connect 0x7fb54c004870 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb55bfff700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 0x7fb55c133bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c750 tx=0x7fb54800ca60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb5597fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548007250 con 0x7fb54c004870 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.319+0000 7fb5597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: dump: 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.333 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 5 monclient: authenticate success, global_id 94168 2024-09-16T11:04:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480073d0 con 0x7fb54c004870 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _renew_subs 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548004820 con 0x7fb54c004870 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb55affd700 4 set_mon_vals no callback set 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c122210 con 0x7fb54c004870 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 monclient: dump: 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.335 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 client.94168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _renew_subs 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c09b130 con 0x7fb54c004870 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 1 librados: init done 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c09fef0 con 0x7fb54c004870 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb548004980 con 0x7fb54c004870 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 1 --2- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 0x7fb540039be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(576..576 src has 395..576) v4 ==== 5374+0+0 (secure 0 0 0) 0x7fb548046f80 con 0x7fb54c004870 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 client.94168.objecter ms_dispatch 0x7fb55c000fd0 osd_map(576..576 src has 395..576) v4 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 3 client.94168.objecter handle_osd_map got epochs [576,576] > 0 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 3 client.94168.objecter handle_osd_map decoding full epoch 576 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 20 client.94168.objecter dump_active .. 0 homeless 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb560e08700 1 --2- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 0x7fb540039be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb560e08700 10 monclient: get_auth_request con 0x7fb540037720 auth_method 0 2024-09-16T11:04:12.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb560e08700 1 --2- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 0x7fb540039be0 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7fb550007dd0 tx=0x7fb550009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.323+0000 7fb5597fa700 10 client.94168.objecter ms_handle_connect 0x7fb540037720 2024-09-16T11:04:12.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.335+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb548011720 con 0x7fb54c004870 2024-09-16T11:04:12.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.335+0000 7fb5597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.335+0000 7fb5597fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:04:12.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.563+0000 7fb561e0a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:04:12.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.563+0000 7fb561e0a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:12.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.563+0000 7fb561e0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:04:12.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.563+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb55c1346a0 con 0x7fb54c004870 2024-09-16T11:04:12.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb5597fa700 1 -- 172.21.15.5:0/3542714751 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7fb55c1346a0 con 0x7fb54c004870 2024-09-16T11:04:12.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb5597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:04:12.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb5597fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:04:12.570 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 10 librados: watch_flush enter 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 10 librados: watch_flush exit 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 20 client.94168.objecter shutdown clearing up homeless session... 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 10 client.94168.objecter successfully canceled tick 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 msgr2=0x7fb540039be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 0x7fb540039be0 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7fb550007dd0 tx=0x7fb550009040 comp rx=0 tx=0).stop 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 10 monclient: shutdown 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 msgr2=0x7fb55c133bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 0x7fb55c133bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb54800c750 tx=0x7fb54800ca60 comp rx=0 tx=0).stop 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 shutdown_connections 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb540037720 0x7fb540039be0 unknown :-1 s=CLOSED pgs=3877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb54c005280 0x7fb55c134130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb54c004870 0x7fb55c133bf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 --2- 172.21.15.5:0/3542714751 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb54c003a20 0x7fb55c1c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 >> 172.21.15.5:0/3542714751 conn(0x7fb55c0a45d0 msgr2=0x7fb55c11a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 shutdown_connections 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 -- 172.21.15.5:0/3542714751 wait complete. 2024-09-16T11:04:12.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.567+0000 7fb561e0a700 1 librados: shutdown 2024-09-16T11:04:12.585 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T11:04:12.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T11:04:12.586 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: get_monmap_and_config 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: build_initial_monmap 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: monmap: 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:04:12.704926+0000 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:04:12.704926+0000 2024-09-16T11:04:12.706 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:04:12.707 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 Processor -- start 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 -- start start 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: init 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c480a0900 addr v1:172.21.15.5:6789/0 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2c480a13f0 0x7f2c4809abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-a con 0x7f2c480a13f0 addr v2:172.21.15.5:3300/0 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-b con 0x7f2c4813aba0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c48073c20 con 0x7f2c480a0900 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c48071d60 con 0x7f2c480a13f0 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4809a1b0 con 0x7f2c4813aba0 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient(hunting): authenticate will time out at 1711.281613s 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4e4ac700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 msgr2=0x7f2c4813cf90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4dcab700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2c480a0900 0x7f2c480a0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33670/0 (socket says 172.21.15.5:33670) 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4dcab700 1 -- 172.21.15.5:0/3188103589 learned_addr learned my addr 172.21.15.5:0/3188103589 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4e4ac700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4813cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:12.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/3188103589 >> v2:172.21.15.5:3300/0 conn(0x7f2c480a13f0 0x7f2c4809abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request con 0x7f2c480a13f0 auth_method 0 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 1 -- 172.21.15.5:0/3188103589 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404042359 0 0) 0x7f2c48073c20 con 0x7f2c480a0900 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient(hunting): _negotiate 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient(hunting): my global_id is 74487 2024-09-16T11:04:12.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 1 -- 172.21.15.5:0/3188103589 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c340035d0 con 0x7f2c480a0900 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_done global_id 74484 payload 274 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 -- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 msgr2=0x7f2c4813cf90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4813cf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 -- 172.21.15.5:0/3188103589 >> v1:172.21.15.5:6789/0 conn(0x7f2c480a0900 legacy=0x7f2c480a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 monclient: found mon.noname-a 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 -- 172.21.15.5:0/3188103589 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4809a750 con 0x7f2c480a13f0 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a13f0 0x7f2c4809abe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009850 tx=0x7f2c38009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae5dcaa265dc880 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 1 -- 172.21.15.5:0/3188103589 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38011830 con 0x7f2c480a13f0 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: dump: 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 5 monclient: authenticate success, global_id 74484 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 1 -- 172.21.15.5:0/3188103589 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c38011990 con 0x7f2c480a13f0 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: get_monmap_and_config success 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 4 set_mon_vals no callback set 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 1 -- 172.21.15.5:0/3188103589 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38011d00 con 0x7f2c480a13f0 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4cca9700 10 monclient: dump: 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.716 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 10 monclient: shutdown 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 -- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a13f0 msgr2=0x7f2c4809abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.703+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a13f0 0x7f2c4809abe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009850 tx=0x7f2c38009b60 comp rx=0 tx=0).stop 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- 172.21.15.5:0/3188103589 shutdown_connections 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4813cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/3188103589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a13f0 0x7f2c4809abe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- 172.21.15.5:0/3188103589 >> 172.21.15.5:0/3188103589 conn(0x7f2c4806a4f0 msgr2=0x7f2c4806a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- 172.21.15.5:0/3188103589 shutdown_connections 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- 172.21.15.5:0/3188103589 wait complete. 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient: build_initial_monmap 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient: monmap: 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:04:12.718 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:04:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:04:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:04:12.719 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:04:12.719 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 librados: starting msgr at 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 librados: starting objecter 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 Processor -- start 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- start start 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 librados: setting wanted keys 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 librados: calling monclient init 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient: init 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-a con 0x7f2c480a0900 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:12.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c480a13f0 0x7f2c48135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-b con 0x7f2c480a13f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4811c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): picked mon.noname-c con 0x7f2c4813aba0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c481cacc0 con 0x7f2c480a0900 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4811f380 con 0x7f2c480a13f0 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4813b110 con 0x7f2c4813aba0 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4ecad700 10 monclient(hunting): authenticate will time out at 1711.286503s 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4e4ac700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 msgr2=0x7f2c4811c9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:12.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4e4ac700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4811c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48266/0 (socket says 172.21.15.5:48266) 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 -- 172.21.15.5:0/2675395787 learned_addr learned my addr 172.21.15.5:0/2675395787 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): get_auth_request con 0x7f2c480a0900 auth_method 0 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c480a13f0 0x7f2c48135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request con 0x7f2c480a13f0 auth_method 0 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient(hunting): handle_auth_done global_id 74490 payload 824 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 msgr2=0x7f2c4811c9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2c4813aba0 0x7f2c4811c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c480a13f0 msgr2=0x7f2c48135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c480a13f0 0x7f2c48135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 monclient: found mon.noname-a 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c380094b0 con 0x7f2c480a0900 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 10 client.?.objecter ms_handle_connect 0x7f2c480a0900 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c4dcab700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c4f0 tx=0x7f2c4400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44005d90 con 0x7f2c480a0900 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.707+0000 7f2c3e7fc700 10 monclient: dump: 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.731 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 5 monclient: authenticate success, global_id 74490 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 monclient: _renew_subs 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c440070d0 con 0x7f2c480a0900 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4809c0e0 con 0x7f2c480a0900 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44007440 con 0x7f2c480a0900 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3ffff700 4 set_mon_vals no callback set 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 client.74490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: dump: 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 monclient: _renew_subs 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4811fdb0 con 0x7f2c480a0900 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 1 librados: init done 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 0x7f2c4811cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48067430 con 0x7f2c4811ff40 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 0x7f2c4811cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient: get_auth_request con 0x7f2c4811ff40 auth_method 0 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_done global_id 74493 payload 824 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c480a4d20 con 0x7f2c4811ff40 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2c440075a0 con 0x7f2c480a0900 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 0x7f2c4811cef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009850 tx=0x7f2c38004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 --2- 172.21.15.5:0/2675395787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c2c036f10 0x7f2c2c0393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 client.74490.objecter ms_handle_connect 0x7f2c4811ff40 2024-09-16T11:04:12.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 client.74490.objecter resend_mon_ops 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(576..576 src has 395..576) v4 ==== 5374+0+0 (secure 0 0 0) 0x7f2c44047200 con 0x7f2c480a0900 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 client.74490.objecter ms_dispatch 0x7f2c48000fd0 osd_map(576..576 src has 395..576) v4 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 3 client.74490.objecter handle_osd_map got epochs [576,576] > 0 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 3 client.74490.objecter handle_osd_map decoding full epoch 576 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 20 client.74490.objecter dump_active .. 0 homeless 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38020070 con 0x7f2c4811ff40 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4e4ac700 1 --2- 172.21.15.5:0/2675395787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c2c036f10 0x7f2c2c0393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: dump: 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.736 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4e4ac700 10 monclient: get_auth_request con 0x7f2c2c036f10 auth_method 0 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2c38011cc0 con 0x7f2c4811ff40 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:04:12.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 msgr2=0x7f2c4811cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 0x7f2c4811cef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009850 tx=0x7f2c38004770 comp rx=0 tx=0).stop 2024-09-16T11:04:12.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 msgr2=0x7f2c4811cef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:04:12.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c4811ff40 0x7f2c4811cef0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c4e4ac700 1 --2- 172.21.15.5:0/2675395787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c2c036f10 0x7f2c2c0393d0 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f2c4811ede0 tx=0x7f2c40009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.711+0000 7f2c3e7fc700 10 client.74490.objecter ms_handle_connect 0x7f2c2c036f10 2024-09-16T11:04:12.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:04:12.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 0x7f2c4811d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4813d4d0 con 0x7f2c481361e0 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 0x7f2c4811d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient: get_auth_request con 0x7f2c481361e0 auth_method 0 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 10 monclient(hunting): handle_auth_done global_id 74496 payload 824 2024-09-16T11:04:12.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 1 -- 172.21.15.5:0/2675395787 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2c4809abe0 con 0x7f2c481361e0 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 client.74490.objecter ms_handle_connect 0x7f2c481361e0 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 client.74490.objecter resend_mon_ops 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4d4aa700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 0x7f2c4811d430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009ba0 tx=0x7f2c38011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3801b840 con 0x7f2c481361e0 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 monclient: dump: 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.741 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4768 (secure 0 0 0) 0x7f2c38003e50 con 0x7f2c481361e0 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:04:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 msgr2=0x7f2c4811d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 0x7f2c4811d430 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009ba0 tx=0x7f2c38011000 comp rx=0 tx=0).stop 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 msgr2=0x7f2c4811d430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c3e7fc700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c481361e0 0x7f2c4811d430 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009ba0 tx=0x7f2c38011000 comp rx=0 tx=0).stop 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 58, 2024-09-16T11:04:12.743 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.744 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.745 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:04:12.746 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.747 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.748 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:04:12.749 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:04:12.750 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.751 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.752 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:04:12.753 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:04:12.754 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 5 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:04:12.755 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.756 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.757 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:04:12.757 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:04:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 librados: watch_flush enter 2024-09-16T11:04:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 librados: watch_flush exit 2024-09-16T11:04:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 20 client.74490.objecter shutdown clearing up homeless session... 2024-09-16T11:04:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 client.74490.objecter successfully canceled tick 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c2c036f10 msgr2=0x7f2c2c0393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/2675395787 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2c2c036f10 0x7f2c2c0393d0 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f2c4811ede0 tx=0x7f2c40009270 comp rx=0 tx=0).stop 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 10 monclient: shutdown 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 msgr2=0x7f2c481ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c4f0 tx=0x7f2c4400c800 comp rx=0 tx=0).stop 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4e4ac700 1 -- 172.21.15.5:0/2675395787 reap_dead start 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 shutdown_connections 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 --2- 172.21.15.5:0/2675395787 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2c480a0900 0x7f2c481ca780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 >> 172.21.15.5:0/2675395787 conn(0x7f2c4806a4f0 msgr2=0x7f2c480a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 shutdown_connections 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 -- 172.21.15.5:0/2675395787 wait complete. 2024-09-16T11:04:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.723+0000 7f2c4ecad700 1 librados: shutdown 2024-09-16T11:04:12.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:04:12.760 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T11:04:12.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:04:12.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.871+0000 7fe363281700 10 monclient: get_monmap_and_config 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.871+0000 7fe363281700 10 monclient: build_initial_monmap 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.871+0000 7fe363281700 10 monclient: monmap: 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:04:12.876161+0000 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:04:12.876161+0000 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:04:12.878 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:04:12.879 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.871+0000 7fe363281700 1 Processor -- start 2024-09-16T11:04:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- start start 2024-09-16T11:04:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: init 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe35c0a2bf0 0x7fe35c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): picked mon.noname-c con 0x7fe35c0a2bf0 addr v2:172.21.15.143:3300/0 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe35c09f780 addr v1:172.21.15.143:6789/0 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe35c0a3510 0x7fe35c1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): picked mon.noname-a con 0x7fe35c0a3510 addr v2:172.21.15.5:3300/0 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35c0a3c40 con 0x7fe35c09f780 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35c0a4300 con 0x7fe35c0a3510 2024-09-16T11:04:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35c09fda0 con 0x7fe35c0a2bf0 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient(hunting): authenticate will time out at 1711.452633s 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe35c0a3510 0x7fe35c1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe35c0a3510 0x7fe35c1360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48300/0 (socket says 172.21.15.5:48300) 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 -- 172.21.15.5:0/306034275 learned_addr learned my addr 172.21.15.5:0/306034275 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): get_auth_request con 0x7fe35c0a3510 auth_method 0 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36227f700 1 --2- 172.21.15.5:0/306034275 >> v2:172.21.15.143:3300/0 conn(0x7fe35c0a2bf0 0x7fe35c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36227f700 10 monclient(hunting): get_auth_request con 0x7fe35c0a2bf0 auth_method 0 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36227f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36227f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36227f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient(hunting): handle_auth_done global_id 74499 payload 274 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 -- 172.21.15.5:0/306034275 >> v2:172.21.15.143:3300/0 conn(0x7fe35c0a2bf0 msgr2=0x7fe35c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 --2- 172.21.15.5:0/306034275 >> v2:172.21.15.143:3300/0 conn(0x7fe35c0a2bf0 0x7fe35c0a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 -- 172.21.15.5:0/306034275 >> v1:172.21.15.143:6789/0 conn(0x7fe35c09f780 legacy=0x7fe35c1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 monclient: found mon.noname-a 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 -- 172.21.15.5:0/306034275 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0a0080 con 0x7fe35c0a3510 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe362a80700 1 --2- 172.21.15.5:0/306034275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c0a3510 0x7fe35c1360c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe35c074960 tx=0x7fe35400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ef539166f359ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 1 -- 172.21.15.5:0/306034275 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400e070 con 0x7fe35c0a3510 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: dump: 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.891 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 1 -- 172.21.15.5:0/306034275 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe354003a40 con 0x7fe35c0a3510 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 5 monclient: authenticate success, global_id 74499 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: get_monmap_and_config success 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 4 set_mon_vals no callback set 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 1 -- 172.21.15.5:0/306034275 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe354003df0 con 0x7fe35c0a3510 2024-09-16T11:04:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe36127d700 10 monclient: dump: 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.893 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: shutdown 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- 172.21.15.5:0/306034275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c0a3510 msgr2=0x7fe35c1360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 --2- 172.21.15.5:0/306034275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c0a3510 0x7fe35c1360c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe35c074960 tx=0x7fe35400b330 comp rx=0 tx=0).stop 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- 172.21.15.5:0/306034275 shutdown_connections 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 --2- 172.21.15.5:0/306034275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c0a3510 0x7fe35c1360c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 --2- 172.21.15.5:0/306034275 >> v2:172.21.15.143:3300/0 conn(0x7fe35c0a2bf0 0x7fe35c0a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- 172.21.15.5:0/306034275 >> 172.21.15.5:0/306034275 conn(0x7fe35c0a53e0 msgr2=0x7fe35c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- 172.21.15.5:0/306034275 shutdown_connections 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- 172.21.15.5:0/306034275 wait complete. 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: build_initial_monmap 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: monmap: 2024-09-16T11:04:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 librados: starting msgr at 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 librados: starting objecter 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 Processor -- start 2024-09-16T11:04:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 -- start start 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 librados: setting wanted keys 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 1 librados: calling monclient init 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.875+0000 7fe363281700 10 monclient: init 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _reopen_session rank -1 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): picked mon.noname-a con 0x7fe35c09f780 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe35c0a2bf0 0x7fe35c123590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): picked mon.noname-c con 0x7fe35c0a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c0a3510 0x7fe35c123ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): picked mon.noname-b con 0x7fe35c0a3510 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c135b20 con 0x7fe35c09f780 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c09f4e0 con 0x7fe35c0a3510 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c070650 con 0x7fe35c0a2bf0 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): _renew_subs 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): authenticate will time out at 1711.456454s 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe35c0a2bf0 msgr2=0x7fe35c123590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe35c0a2bf0 0x7fe35c123590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48316/0 (socket says 172.21.15.5:48316) 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 -- 172.21.15.5:0/2975604662 learned_addr learned my addr 172.21.15.5:0/2975604662 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): get_auth_request con 0x7fe35c09f780 auth_method 0 2024-09-16T11:04:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c0a3510 0x7fe35c123ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): get_auth_request con 0x7fe35c0a3510 auth_method 0 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient(hunting): handle_auth_done global_id 74502 payload 824 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe35c0a2bf0 msgr2=0x7fe35c123590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe35c0a2bf0 0x7fe35c123590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c0a3510 msgr2=0x7fe35c123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c0a3510 0x7fe35c123ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient: _finish_hunting 0 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 monclient: found mon.noname-a 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009b30 con 0x7fe35c09f780 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 client.?.objecter ms_handle_connect 0x7fe35c09f780 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:04:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe36227f700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe35800e810 tx=0x7fe35800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358013610 con 0x7fe35c09f780 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: dump: 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: _finish_auth 0 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 5 monclient: authenticate success, global_id 74502 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358013770 con 0x7fe35c09f780 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _renew_subs 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c126d10 con 0x7fe35c09f780 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358013b20 con 0x7fe35c09f780 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe353fff700 4 set_mon_vals no callback set 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: dump: 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 client.74502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _renew_subs 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c124940 con 0x7fe35c09f780 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 librados: init done 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 0x7fe35c11ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c074d50 con 0x7fe35c11e840 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe358013c80 con 0x7fe35c09f780 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 0x7fe35c11ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient: get_auth_request con 0x7fe35c11e840 auth_method 0 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 --2- 172.21.15.5:0/2975604662 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe340036f10 0x7fe3400393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(576..576 src has 395..576) v4 ==== 5374+0+0 (secure 0 0 0) 0x7fe358047a40 con 0x7fe35c09f780 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 client.74502.objecter ms_dispatch 0x7fe35c000fd0 osd_map(576..576 src has 395..576) v4 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 3 client.74502.objecter handle_osd_map got epochs [576,576] > 0 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 3 client.74502.objecter handle_osd_map decoding full epoch 576 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 10 monclient(hunting): handle_auth_done global_id 94180 payload 824 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe35c0a3960 con 0x7fe35c11e840 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 20 client.74502.objecter dump_active .. 0 homeless 2024-09-16T11:04:12.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 1 --2- 172.21.15.5:0/2975604662 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe340036f10 0x7fe3400393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe362a80700 10 monclient: get_auth_request con 0x7fe340036f10 auth_method 0 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 client.74502.objecter ms_handle_connect 0x7fe35c11e840 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 client.74502.objecter resend_mon_ops 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe361a7e700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 0x7fe35c11ec20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe35c06bae0 tx=0x7fe34c00c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c005c80 con 0x7fe35c11e840 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.879+0000 7fe352ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 10 monclient: dump: 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.905 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 10 client.74502.objecter ms_handle_connect 0x7fe340036f10 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe362a80700 1 --2- 172.21.15.5:0/2975604662 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe340036f10 0x7fe3400393d0 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fe354015040 tx=0x7fe354009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe34c005de0 con 0x7fe35c11e840 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 msgr2=0x7fe35c11ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 0x7fe35c11ec20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe35c06bae0 tx=0x7fe34c00c710 comp rx=0 tx=0).stop 2024-09-16T11:04:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 msgr2=0x7fe35c11ec20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:04:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.883+0000 7fe352ffd700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11e840 0x7fe35c11ec20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe363281700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe363281700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 0x7fe35c1d4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe363281700 10 monclient(hunting): start opening mon connection 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c0a7960 con 0x7fe35c11f160 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 0x7fe35c1d4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient: get_auth_request con 0x7fe35c11f160 auth_method 0 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:04:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 10 monclient(hunting): handle_auth_done global_id 94183 payload 824 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 1 -- 172.21.15.5:0/2975604662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe35c0a2fd0 con 0x7fe35c11f160 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe352ffd700 10 client.74502.objecter ms_handle_connect 0x7fe35c11f160 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe352ffd700 10 client.74502.objecter resend_mon_ops 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.891+0000 7fe361a7e700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 0x7fe35c1d4c30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe34c014000 tx=0x7fe34c003860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c01f070 con 0x7fe35c11f160 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 10 monclient: dump: 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:04:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4600 (secure 0 0 0) 0x7fe34c012400 con 0x7fe35c11f160 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 msgr2=0x7fe35c1d4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 0x7fe35c1d4c30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe34c014000 tx=0x7fe34c003860 comp rx=0 tx=0).stop 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 msgr2=0x7fe35c1d4c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:04:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe352ffd700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe35c11f160 0x7fe35c1d4c30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 58, 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.912 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.913 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.914 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:04:12.915 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:04:12.916 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.917 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.918 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.919 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.920 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:04:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:04:12.922 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.923 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:04:12.924 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:04:12.925 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:04:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 10 librados: watch_flush enter 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 10 librados: watch_flush exit 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 20 client.74502.objecter shutdown clearing up homeless session... 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 10 client.74502.objecter successfully canceled tick 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe340036f10 msgr2=0x7fe3400393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 --2- 172.21.15.5:0/2975604662 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe340036f10 0x7fe3400393d0 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fe354015040 tx=0x7fe354009000 comp rx=0 tx=0).stop 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 10 monclient: shutdown 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 msgr2=0x7fe35c123050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe362a80700 1 -- 172.21.15.5:0/2975604662 reap_dead start 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe35800e810 tx=0x7fe35800eb20 comp rx=0 tx=0).stop 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 shutdown_connections 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 --2- 172.21.15.5:0/2975604662 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe35c09f780 0x7fe35c123050 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:04:12.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 >> 172.21.15.5:0/2975604662 conn(0x7fe35c0a53e0 msgr2=0x7fe35c06fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:04:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 shutdown_connections 2024-09-16T11:04:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 -- 172.21.15.5:0/2975604662 wait complete. 2024-09-16T11:04:12.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:04:12.895+0000 7fe363281700 1 librados: shutdown 2024-09-16T11:04:12.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:04:12.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T11:05:42.929 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T11:05:42.929 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T11:05:42.930 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T11:05:42.930 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T11:05:42.934 INFO:tasks.ceph.mon.c:Started 2024-09-16T11:05:42.934 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:05:42.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:05:43.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 10 monclient: get_monmap_and_config 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 10 monclient: build_initial_monmap 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 10 monclient: monmap: 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:43.039625+0000 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:43.039625+0000 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:43.042 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 1 Processor -- start 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 1 -- start start 2024-09-16T11:05:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.033+0000 7f667f995700 10 monclient: init 2024-09-16T11:05:43.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:43.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T11:05:43.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f667809d6d0 0x7f667809dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:43.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): picked mon.noname-c con 0x7f667809d6d0 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:43.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f667809ea30 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66780a0620 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667809fbb0 con 0x7f66780a0620 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6678134630 con 0x7f667809ea30 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667809adf0 con 0x7f667809d6d0 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient(hunting): authenticate will time out at 1801.616221s 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667df32700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f66780a0620 0x7f6678133f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57042/0 (socket says 172.21.15.5:57042) 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667df32700 1 -- 172.21.15.5:0/2194015930 learned_addr learned my addr 172.21.15.5:0/2194015930 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 --2- 172.21.15.5:0/2194015930 >> v2:172.21.15.143:3300/0 conn(0x7f667809d6d0 0x7f667809dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:43.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): get_auth_request con 0x7f667809d6d0 auth_method 0 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 1 -- 172.21.15.5:0/2194015930 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539126523 0 0) 0x7f667809fbb0 con 0x7f66780a0620 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient(hunting): _negotiate 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient(hunting): my global_id is 74505 2024-09-16T11:05:43.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 1 -- 172.21.15.5:0/2194015930 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66600035d0 con 0x7f66780a0620 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient(hunting): handle_auth_done global_id 94186 payload 274 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 -- 172.21.15.5:0/2194015930 >> v1:172.21.15.143:6789/0 conn(0x7f667809ea30 legacy=0x7f6678133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 -- 172.21.15.5:0/2194015930 >> v1:172.21.15.5:6789/0 conn(0x7f66780a0620 legacy=0x7f6678133f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient: _finish_hunting 0 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 monclient: found mon.noname-c 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 -- 172.21.15.5:0/2194015930 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667809af80 con 0x7f667809d6d0 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667d731700 1 --2- 172.21.15.5:0/2194015930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f667809dab0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6668009d90 tx=0x7f666800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda22fdbcf62394d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 1 -- 172.21.15.5:0/2194015930 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800e070 con 0x7f667809d6d0 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:05:43.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: dump: 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: _finish_auth 0 2024-09-16T11:05:43.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: _check_auth_tickets 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 5 monclient: authenticate success, global_id 94186 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 1 -- 172.21.15.5:0/2194015930 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666800bb20 con 0x7f667809d6d0 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 4 set_mon_vals no callback set 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 1 -- 172.21.15.5:0/2194015930 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666800be90 con 0x7f667809d6d0 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f6677fff700 10 monclient: dump: 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:43.056 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: shutdown 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:43.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- 172.21.15.5:0/2194015930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 msgr2=0x7f667809dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 --2- 172.21.15.5:0/2194015930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f667809dab0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6668009d90 tx=0x7f666800b330 comp rx=0 tx=0).stop 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- 172.21.15.5:0/2194015930 shutdown_connections 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 --2- 172.21.15.5:0/2194015930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f667809dab0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- 172.21.15.5:0/2194015930 >> 172.21.15.5:0/2194015930 conn(0x7f66780a0b30 msgr2=0x7f66780a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- 172.21.15.5:0/2194015930 shutdown_connections 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- 172.21.15.5:0/2194015930 wait complete. 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: build_initial_monmap 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: monmap: 2024-09-16T11:05:43.058 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 librados: starting msgr at 2024-09-16T11:05:43.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 librados: starting objecter 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 Processor -- start 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 -- start start 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 librados: setting wanted keys 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 1 librados: calling monclient init 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.037+0000 7f667f995700 10 monclient: init 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f6678132cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): picked mon.noname-b con 0x7f667809d6d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:43.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): picked mon.noname-a con 0x7f667809ea30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 0x7f667811c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): picked mon.noname-c con 0x7f66781c8560 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66781c92c0 con 0x7f667809ea30 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667811fdd0 con 0x7f667809d6d0 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6678133730 con 0x7f66781c8560 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient(hunting): authenticate will time out at 1801.620184s 2024-09-16T11:05:43.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667df32700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 msgr2=0x7f667811c670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667df32700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 0x7f667811c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35430/0 (socket says 172.21.15.5:35430) 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 -- 172.21.15.5:0/1434063437 learned_addr learned my addr 172.21.15.5:0/1434063437 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): get_auth_request con 0x7f667809ea30 auth_method 0 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:43.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f6678132cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 10 monclient(hunting): get_auth_request con 0x7f667809d6d0 auth_method 0 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient(hunting): handle_auth_done global_id 74508 payload 824 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 -- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 msgr2=0x7f667811c670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 0x7f667811c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 -- 172.21.15.5:0/1434063437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 msgr2=0x7f6678132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f6678132cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient: _finish_hunting 0 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 monclient: found mon.noname-a 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 -- 172.21.15.5:0/1434063437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6668009a70 con 0x7f667809ea30 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667cf30700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c500 tx=0x7f666c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 client.?.objecter ms_handle_connect 0x7f667809ea30 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c0052d0 con 0x7f667809ea30 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:43.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: dump: 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: _finish_auth 0 2024-09-16T11:05:43.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c005430 con 0x7f667809ea30 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 5 monclient: authenticate success, global_id 74508 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666c0055a0 con 0x7f667809ea30 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 monclient: dump: 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.066 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f66777fe700 4 set_mon_vals no callback set 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _renew_subs 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6678120670 con 0x7f667809ea30 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 client.74508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:43.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _renew_subs 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667806cb50 con 0x7f667809ea30 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 librados: init done 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f666c0036a0 con 0x7f667809ea30 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 --2- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 0x7f66640393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(651..651 src has 395..651) v4 ==== 5809+0+0 (secure 0 0 0) 0x7f666c047390 con 0x7f667809ea30 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 client.74508.objecter ms_dispatch 0x7f6678000fd0 osd_map(651..651 src has 395..651) v4 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 3 client.74508.objecter handle_osd_map got epochs [651,651] > 0 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 3 client.74508.objecter handle_osd_map decoding full epoch 651 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 20 client.74508.objecter dump_active .. 0 homeless 2024-09-16T11:05:43.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 1 --2- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 0x7f66640393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 10 monclient: get_auth_request con 0x7f6664036f10 auth_method 0 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667811ff10 con 0x7f667809ea30 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f6675ffb700 10 client.74508.objecter ms_handle_connect 0x7f6664036f10 2024-09-16T11:05:43.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.041+0000 7f667d731700 1 --2- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 0x7f66640393d0 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7f666800b370 tx=0x7f6668011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:43.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.053+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f667811ff10 con 0x7f667809ea30 2024-09-16T11:05:43.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.053+0000 7f6675ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:43.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.053+0000 7f6675ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:43.070 INFO:tasks.ceph.mon.c.smithi005.stderr:2024-09-16T11:05:43.053+0000 7fbaee84fd40 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 607 -> 0 (sync_force? mon rebuild?) 2024-09-16T11:05:43.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f667f995700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f667f995700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f667f995700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f667811ff10 con 0x7f667809ea30 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f6675ffb700 1 -- 172.21.15.5:0/1434063437 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f666c03a610 con 0x7f667809ea30 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f6675ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:43.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.281+0000 7f6675ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:43.289 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:05:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 10 librados: watch_flush enter 2024-09-16T11:05:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 10 librados: watch_flush exit 2024-09-16T11:05:43.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 20 client.74508.objecter shutdown clearing up homeless session... 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 10 client.74508.objecter successfully canceled tick 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 msgr2=0x7f66640393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 0x7f66640393d0 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7f666800b370 tx=0x7f6668011000 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 10 monclient: shutdown 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 msgr2=0x7f66781331f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c500 tx=0x7f666c00c8c0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 shutdown_connections 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f66781c8560 0x7f667811c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6664036f10 0x7f66640393d0 unknown :-1 s=CLOSED pgs=3887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f667809ea30 0x7f66781331f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 --2- 172.21.15.5:0/1434063437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f667809d6d0 0x7f6678132cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:43.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 >> 172.21.15.5:0/1434063437 conn(0x7f66780a0b30 msgr2=0x7f66780a2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 shutdown_connections 2024-09-16T11:05:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 -- 172.21.15.5:0/1434063437 wait complete. 2024-09-16T11:05:43.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:43.285+0000 7f667f995700 1 librados: shutdown 2024-09-16T11:05:46.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:05:46.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: get_monmap_and_config 2024-09-16T11:05:46.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: build_initial_monmap 2024-09-16T11:05:46.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: monmap: 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:46.419080+0000 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:46.419080+0000 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:46.421 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 Processor -- start 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 -- start start 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: init 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f713009e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f713009d6a0 0x7f7130138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): picked mon.noname-c con 0x7f713009d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): picked mon.noname-b con 0x7f713009dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7130070e40 con 0x7f713009e8e0 2024-09-16T11:05:46.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71300a01d0 con 0x7f713009d6a0 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713009fef0 con 0x7f713009dfd0 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f713547b700 10 monclient(hunting): authenticate will time out at 1804.995635s 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f7134c7a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f7134c7a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40546/0 (socket says 172.21.15.5:40546) 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.413+0000 7f7134c7a700 1 -- 172.21.15.5:0/3647369153 learned_addr learned my addr 172.21.15.5:0/3647369153 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:46.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 10 monclient(hunting): get_auth_request con 0x7f713009dfd0 auth_method 0 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 --2- 172.21.15.5:0/3647369153 >> v2:172.21.15.143:3300/0 conn(0x7f713009d6a0 0x7f7130138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 1 -- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 msgr2=0x7f713013b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 1 -- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 msgr2=0x7f713013b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): get_auth_request con 0x7f713009d6a0 auth_method 0 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f7134c7a700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:46.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 1 -- 172.21.15.5:0/3647369153 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153795357 0 0) 0x7f7130070e40 con 0x7f713009e8e0 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient(hunting): _negotiate 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient(hunting): my global_id is 94198 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 1 -- 172.21.15.5:0/3647369153 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71180035d0 con 0x7f713009e8e0 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient(hunting): handle_auth_done global_id 94195 payload 274 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 -- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 msgr2=0x7f713013b0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 -- 172.21.15.5:0/3647369153 >> v1:172.21.15.143:6789/0 conn(0x7f713009e8e0 legacy=0x7f713009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 monclient: found mon.noname-c 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 -- 172.21.15.5:0/3647369153 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713009f9f0 con 0x7f713009d6a0 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712f7fe700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009d6a0 0x7f7130138b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7120009880 tx=0x7f7120009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5607d6ce61312209 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 1 -- 172.21.15.5:0/3647369153 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120011830 con 0x7f713009d6a0 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: dump: 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:46.430 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: _finish_auth 0 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: _check_auth_tickets 2024-09-16T11:05:46.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 1 -- 172.21.15.5:0/3647369153 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120011990 con 0x7f713009d6a0 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 1 -- 172.21.15.5:0/3647369153 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120011d20 con 0x7f713009d6a0 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f712effd700 10 monclient: dump: 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:46.432 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 5 monclient: authenticate success, global_id 94195 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 4 set_mon_vals no callback set 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 10 monclient: shutdown 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 -- 172.21.15.5:0/3647369153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009d6a0 msgr2=0x7f7130138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009d6a0 0x7f7130138b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7120009880 tx=0x7f7120009b90 comp rx=0 tx=0).stop 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 -- 172.21.15.5:0/3647369153 shutdown_connections 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009dfd0 0x7f713013b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 --2- 172.21.15.5:0/3647369153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009d6a0 0x7f7130138b80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 -- 172.21.15.5:0/3647369153 >> 172.21.15.5:0/3647369153 conn(0x7f71300a45d0 msgr2=0x7f71300a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 -- 172.21.15.5:0/3647369153 shutdown_connections 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 -- 172.21.15.5:0/3647369153 wait complete. 2024-09-16T11:05:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 10 monclient: build_initial_monmap 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 10 monclient: monmap: 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 librados: starting msgr at 2024-09-16T11:05:46.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.417+0000 7f713547b700 1 librados: starting objecter 2024-09-16T11:05:46.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 Processor -- start 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- start start 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 librados: setting wanted keys 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 librados: calling monclient init 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: init 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): picked mon.noname-c con 0x7f713009d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): picked mon.noname-a con 0x7f713009dfd0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 0x7f7130133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): picked mon.noname-b con 0x7f713009e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130070230 con 0x7f713009dfd0 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713011ef50 con 0x7f713009e8e0 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:46.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7130133970 con 0x7f713009d6a0 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient(hunting): authenticate will time out at 1805.001239s 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35472/0 (socket says 172.21.15.5:35472) 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40548/0 (socket says 172.21.15.5:40548) 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 -- 172.21.15.5:0/2585646902 learned_addr learned my addr 172.21.15.5:0/2585646902 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): get_auth_request con 0x7f713009dfd0 auth_method 0 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 0x7f7130133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 10 monclient(hunting): get_auth_request con 0x7f713009d6a0 auth_method 0 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): get_auth_request con 0x7f713009e8e0 auth_method 0 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:46.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 -- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 msgr2=0x7f71301ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 -- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 msgr2=0x7f71301ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712ffff700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient(hunting): handle_auth_done global_id 74511 payload 824 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 -- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 msgr2=0x7f71301ca360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 -- 172.21.15.5:0/2585646902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 msgr2=0x7f7130133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 0x7f7130133430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:05:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 monclient: found mon.noname-a 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 -- 172.21.15.5:0/2585646902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71200094b0 con 0x7f713009dfd0 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f7134c7a700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 0x7f7130133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712f7fe700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7120009bd0 tx=0x7f7120004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 client.?.objecter ms_handle_connect 0x7f713009dfd0 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71200036c0 con 0x7f713009dfd0 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: dump: 2024-09-16T11:05:46.445 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: _finish_auth 0 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: _check_auth_tickets 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 5 monclient: authenticate success, global_id 74511 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120003820 con 0x7f713009dfd0 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _renew_subs 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7130133b20 con 0x7f713009dfd0 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120003bb0 con 0x7f713009dfd0 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712cff9700 10 monclient: dump: 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:46.447 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f712e7fc700 4 set_mon_vals no callback set 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 client.74511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _renew_subs 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713009baf0 con 0x7f713009dfd0 2024-09-16T11:05:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.421+0000 7f713547b700 1 librados: init done 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f713547b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f713547b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f713547b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713009ecc0 con 0x7f713009dfd0 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7120021070 con 0x7f713009dfd0 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 1 --2- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 0x7f711c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(655..655 src has 395..655) v4 ==== 5775+0+0 (secure 0 0 0) 0x7f7120049600 con 0x7f713009dfd0 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 10 client.74511.objecter ms_dispatch 0x7f7130000fd0 osd_map(655..655 src has 395..655) v4 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 3 client.74511.objecter handle_osd_map got epochs [655,655] > 0 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 3 client.74511.objecter handle_osd_map decoding full epoch 655 2024-09-16T11:05:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 20 client.74511.objecter dump_active .. 0 homeless 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712ffff700 1 --2- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 0x7f711c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712ffff700 10 monclient: get_auth_request con 0x7f711c036f60 auth_method 0 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712cff9700 10 client.74511.objecter ms_handle_connect 0x7f711c036f60 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.425+0000 7f712ffff700 1 --2- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 0x7f711c039420 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7f7124007fd0 tx=0x7f7124009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.433+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f712002d3a0 con 0x7f713009dfd0 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.433+0000 7f712cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.433+0000 7f712cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:05:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f71301ca8a0 con 0x7f713009dfd0 2024-09-16T11:05:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f712cff9700 1 -- 172.21.15.5:0/2585646902 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7f7120015080 con 0x7f713009dfd0 2024-09-16T11:05:46.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f712cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:46.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f712cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:46.673 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 librados: watch_flush enter 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 librados: watch_flush exit 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 20 client.74511.objecter shutdown clearing up homeless session... 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 client.74511.objecter successfully canceled tick 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 msgr2=0x7f711c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 0x7f711c039420 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7f7124007fd0 tx=0x7f7124009040 comp rx=0 tx=0).stop 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 10 monclient: shutdown 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 msgr2=0x7f7130132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7120009bd0 tx=0x7f7120004770 comp rx=0 tx=0).stop 2024-09-16T11:05:46.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 shutdown_connections 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f713009e8e0 0x7f7130133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> v2:172.21.15.143:6812/1098205034 conn(0x7f711c036f60 0x7f711c039420 unknown :-1 s=CLOSED pgs=3888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f713009dfd0 0x7f7130132ef0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 --2- 172.21.15.5:0/2585646902 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f713009d6a0 0x7f71301ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.665+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 >> 172.21.15.5:0/2585646902 conn(0x7f71300a45d0 msgr2=0x7f7130134070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.669+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 shutdown_connections 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.669+0000 7f713547b700 1 -- 172.21.15.5:0/2585646902 wait complete. 2024-09-16T11:05:46.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:46.669+0000 7f713547b700 1 librados: shutdown 2024-09-16T11:05:49.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:05:49.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 10 monclient: get_monmap_and_config 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 10 monclient: build_initial_monmap 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 10 monclient: monmap: 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:49.815416+0000 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:49.815416+0000 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:49.817 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:49.818 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:49.818 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 1 Processor -- start 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 1 -- start start 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.808+0000 7fe1859cb700 10 monclient: init 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe18009aa50 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe180071870 0x7fe18009eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-a con 0x7fe180071870 addr v2:172.21.15.5:3300/0 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1801376e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009caa0 con 0x7fe18009aa50 2024-09-16T11:05:49.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18009ae30 con 0x7fe1801376e0 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18009e3e0 con 0x7fe180071870 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient(hunting): authenticate will time out at 1808.392415s 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe180071870 0x7fe18009eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1849c9700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe18009aa50 0x7fe18009e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57058/0 (socket says 172.21.15.5:57058) 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1849c9700 1 -- 172.21.15.5:0/1523542373 learned_addr learned my addr 172.21.15.5:0/1523542373 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): get_auth_request con 0x7fe180071870 auth_method 0 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:49.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient(hunting): handle_auth_done global_id 74514 payload 274 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 -- 172.21.15.5:0/1523542373 >> v1:172.21.15.143:6789/0 conn(0x7fe1801376e0 legacy=0x7fe180139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 1 -- 172.21.15.5:0/1523542373 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378921566 0 0) 0x7fe174001110 con 0x7fe18009aa50 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 -- 172.21.15.5:0/1523542373 >> v1:172.21.15.5:6789/0 conn(0x7fe18009aa50 legacy=0x7fe18009e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient: _finish_hunting 0 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 monclient: found mon.noname-a 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 -- 172.21.15.5:0/1523542373 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18009b5f0 con 0x7fe180071870 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:05:49.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17ffff700 1 --2- 172.21.15.5:0/1523542373 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe180071870 0x7fe18009eff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe18009e520 tx=0x7fe170009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c9805fa6f14b5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 1 -- 172.21.15.5:0/1523542373 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170004230 con 0x7fe180071870 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: dump: 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:49.825 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 1 -- 172.21.15.5:0/1523542373 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe170004390 con 0x7fe180071870 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 5 monclient: authenticate success, global_id 74514 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 1 -- 172.21.15.5:0/1523542373 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1700036c0 con 0x7fe180071870 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 4 set_mon_vals no callback set 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe17f7fe700 10 monclient: dump: 2024-09-16T11:05:49.826 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 10 monclient: shutdown 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- 172.21.15.5:0/1523542373 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe180071870 msgr2=0x7fe18009eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 --2- 172.21.15.5:0/1523542373 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe180071870 0x7fe18009eff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe18009e520 tx=0x7fe170009b30 comp rx=0 tx=0).stop 2024-09-16T11:05:49.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- 172.21.15.5:0/1523542373 shutdown_connections 2024-09-16T11:05:49.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 --2- 172.21.15.5:0/1523542373 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe180071870 0x7fe18009eff0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:49.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- 172.21.15.5:0/1523542373 >> 172.21.15.5:0/1523542373 conn(0x7fe1800a45d0 msgr2=0x7fe1800a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:49.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- 172.21.15.5:0/1523542373 shutdown_connections 2024-09-16T11:05:49.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.812+0000 7fe1859cb700 1 -- 172.21.15.5:0/1523542373 wait complete. 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: build_initial_monmap 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: monmap: 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:49.835 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 librados: starting msgr at 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 librados: starting objecter 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 Processor -- start 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- start start 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 librados: setting wanted keys 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 librados: calling monclient init 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: init 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:05:49.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe180071870 0x7fe18011c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-b con 0x7fe180071870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-c con 0x7fe18009aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 0x7fe18011d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): picked mon.noname-a con 0x7fe1801376e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180120830 con 0x7fe1801376e0 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800a32f0 con 0x7fe180071870 2024-09-16T11:05:49.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18013dff0 con 0x7fe18009aa50 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient(hunting): authenticate will time out at 1808.397308s 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 0x7fe18011d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40556/0 (socket says 172.21.15.5:40556) 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 -- 172.21.15.5:0/203621314 learned_addr learned my addr 172.21.15.5:0/203621314 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 10 monclient(hunting): get_auth_request con 0x7fe18009aa50 auth_method 0 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:49.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): get_auth_request con 0x7fe1801376e0 auth_method 0 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe180071870 0x7fe18011c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 -- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 msgr2=0x7fe18011cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 -- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 msgr2=0x7fe18011cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:49.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): get_auth_request con 0x7fe180071870 auth_method 0 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17ffff700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:49.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1849c9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient(hunting): handle_auth_done global_id 74520 payload 824 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 -- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 msgr2=0x7fe18011cdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 -- 172.21.15.5:0/203621314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe180071870 msgr2=0x7fe18011c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe180071870 0x7fe18011c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient: _finish_hunting 0 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 monclient: found mon.noname-a 2024-09-16T11:05:49.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 -- 172.21.15.5:0/203621314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1700094b0 con 0x7fe1801376e0 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1851ca700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 0x7fe18011d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe17800cb90 tx=0x7fe17800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 client.?.objecter ms_handle_connect 0x7fe1801376e0 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178003c20 con 0x7fe1801376e0 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: dump: 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:49.842 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: _finish_auth 0 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: _check_auth_tickets 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 5 monclient: authenticate success, global_id 74520 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178003d80 con 0x7fe1801376e0 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17800aaa0 con 0x7fe1801376e0 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17d7fa700 10 monclient: dump: 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:49.844 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _renew_subs 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1800a0800 con 0x7fe1801376e0 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe17effd700 4 set_mon_vals no callback set 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 client.74520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _renew_subs 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180120550 con 0x7fe1801376e0 2024-09-16T11:05:49.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.816+0000 7fe1859cb700 1 librados: init done 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1859cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1859cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1859cb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18011ee20 con 0x7fe1801376e0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe17800ac00 con 0x7fe1801376e0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 1 --2- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 0x7fe16c039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(658..658 src has 395..658) v4 ==== 5783+0+0 (secure 0 0 0) 0x7fe178048a50 con 0x7fe1801376e0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 10 client.74520.objecter ms_dispatch 0x7fe180000fd0 osd_map(658..658 src has 395..658) v4 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 3 client.74520.objecter handle_osd_map got epochs [658,658] > 0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 3 client.74520.objecter handle_osd_map decoding full epoch 658 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 20 client.74520.objecter dump_active .. 0 homeless 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1849c9700 1 --2- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 0x7fe16c039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:49.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1849c9700 10 monclient: get_auth_request con 0x7fe16c036f90 auth_method 0 2024-09-16T11:05:49.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe17d7fa700 10 client.74520.objecter ms_handle_connect 0x7fe16c036f90 2024-09-16T11:05:49.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.820+0000 7fe1849c9700 1 --2- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 0x7fe16c039450 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7fe174007fd0 tx=0x7fe174007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:49.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.824+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe1780050d0 con 0x7fe1801376e0 2024-09-16T11:05:49.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.824+0000 7fe17d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:49.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:49.824+0000 7fe17d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:50.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe1859cb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:05:50.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe1859cb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:50.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe1859cb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:50.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe1800710e0 con 0x7fe1801376e0 2024-09-16T11:05:50.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe17d7fa700 1 -- 172.21.15.5:0/203621314 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1530 (secure 0 0 0) 0x7fe178011380 con 0x7fe1801376e0 2024-09-16T11:05:50.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe17d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:50.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.060+0000 7fe17d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:50.070 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 10 librados: watch_flush enter 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 10 librados: watch_flush exit 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 20 client.74520.objecter shutdown clearing up homeless session... 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 10 client.74520.objecter successfully canceled tick 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 msgr2=0x7fe16c039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 0x7fe16c039450 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7fe174007fd0 tx=0x7fe174007e30 comp rx=0 tx=0).stop 2024-09-16T11:05:50.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 10 monclient: shutdown 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 msgr2=0x7fe18011d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 0x7fe18011d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe17800cb90 tx=0x7fe17800cea0 comp rx=0 tx=0).stop 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 shutdown_connections 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe16c036f90 0x7fe16c039450 unknown :-1 s=CLOSED pgs=3889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1801376e0 0x7fe18011d300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe18009aa50 0x7fe18011cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 --2- 172.21.15.5:0/203621314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe180071870 0x7fe18011c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 >> 172.21.15.5:0/203621314 conn(0x7fe1800a45d0 msgr2=0x7fe18013d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 shutdown_connections 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 -- 172.21.15.5:0/203621314 wait complete. 2024-09-16T11:05:50.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:50.064+0000 7fe1859cb700 1 librados: shutdown 2024-09-16T11:05:53.084 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:05:53.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: get_monmap_and_config 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: build_initial_monmap 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: monmap: 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:53.202560+0000 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:53.202560+0000 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:53.204 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:53.205 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:53.205 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:53.205 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:53.205 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 Processor -- start 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- start start 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: init 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9c380a2bf0 0x7f9c380a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-a con 0x7f9c380a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c3809f780 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:53.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c380a3510 0x7f9c381360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-c con 0x7f9c380a3510 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c380a3c40 con 0x7f9c3809f780 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c380a4300 con 0x7f9c380a2bf0 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3809fda0 con 0x7f9c380a3510 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient(hunting): authenticate will time out at 1811.779053s 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9c380a2bf0 0x7f9c380a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c2f7fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9c3809f780 0x7f9c381359b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33010/0 (socket says 172.21.15.5:33010) 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c2f7fe700 1 -- 172.21.15.5:0/2773791051 learned_addr learned my addr 172.21.15.5:0/2773791051 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:53.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): get_auth_request con 0x7f9c380a2bf0 auth_method 0 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 1 --2- 172.21.15.5:0/2773791051 >> v2:172.21.15.143:3300/0 conn(0x7f9c380a3510 0x7f9c381360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): get_auth_request con 0x7f9c380a3510 auth_method 0 2024-09-16T11:05:53.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3ca91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient(hunting): handle_auth_done global_id 74523 payload 274 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 -- 172.21.15.5:0/2773791051 >> v2:172.21.15.143:3300/0 conn(0x7f9c380a3510 msgr2=0x7f9c381360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 1 -- 172.21.15.5:0/2773791051 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710548578 0 0) 0x7f9c380a3c40 con 0x7f9c3809f780 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 --2- 172.21.15.5:0/2773791051 >> v2:172.21.15.143:3300/0 conn(0x7f9c380a3510 0x7f9c381360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 -- 172.21.15.5:0/2773791051 >> v1:172.21.15.5:6789/0 conn(0x7f9c3809f780 legacy=0x7f9c381359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:53.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient: _finish_hunting 0 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 monclient: found mon.noname-a 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 -- 172.21.15.5:0/2773791051 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380a0080 con 0x7f9c380a2bf0 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c37fff700 1 --2- 172.21.15.5:0/2773791051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c380a2fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9c380a0a60 tx=0x7f9c28009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15e915489b7f93c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 1 -- 172.21.15.5:0/2773791051 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28011830 con 0x7f9c380a2bf0 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: dump: 2024-09-16T11:05:53.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: _finish_auth 0 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: _check_auth_tickets 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:53.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 1 -- 172.21.15.5:0/2773791051 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c28011990 con 0x7f9c380a2bf0 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 1 -- 172.21.15.5:0/2773791051 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28011d20 con 0x7f9c380a2bf0 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 5 monclient: authenticate success, global_id 74523 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 4 set_mon_vals no callback set 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c377fe700 10 monclient: dump: 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:53.215 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: shutdown 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- 172.21.15.5:0/2773791051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 msgr2=0x7f9c380a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2773791051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c380a2fd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9c380a0a60 tx=0x7f9c28009b30 comp rx=0 tx=0).stop 2024-09-16T11:05:53.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- 172.21.15.5:0/2773791051 shutdown_connections 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2773791051 >> v2:172.21.15.143:3300/0 conn(0x7f9c380a3510 0x7f9c381360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2773791051 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c380a2fd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- 172.21.15.5:0/2773791051 >> 172.21.15.5:0/2773791051 conn(0x7f9c380a53e0 msgr2=0x7f9c38073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- 172.21.15.5:0/2773791051 shutdown_connections 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- 172.21.15.5:0/2773791051 wait complete. 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: build_initial_monmap 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: monmap: 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:53.217 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:53.218 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:53.218 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:53.218 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 librados: starting msgr at 2024-09-16T11:05:53.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 librados: starting objecter 2024-09-16T11:05:53.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 Processor -- start 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 -- start start 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 librados: setting wanted keys 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 1 librados: calling monclient init 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.200+0000 7f9c3d292700 10 monclient: init 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3809f780 0x7f9c38136580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-b con 0x7f9c3809f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c3811fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-a con 0x7f9c380a2bf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): picked mon.noname-c con 0x7f9c380a3510 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380a7820 con 0x7f9c380a2bf0 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3809f4e0 con 0x7f9c3809f780 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38123cf0 con 0x7f9c380a3510 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient(hunting): authenticate will time out at 1811.784774s 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c3811fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37158/0 (socket says 172.21.15.5:37158) 2024-09-16T11:05:53.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 -- 172.21.15.5:0/2655367012 learned_addr learned my addr 172.21.15.5:0/2655367012 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 10 monclient(hunting): get_auth_request con 0x7f9c380a3510 auth_method 0 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): get_auth_request con 0x7f9c380a2bf0 auth_method 0 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3809f780 0x7f9c38136580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): get_auth_request con 0x7f9c3809f780 auth_method 0 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:53.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 -- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 msgr2=0x7f9c381203a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 -- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 msgr2=0x7f9c381203a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c37fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3ca91700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient(hunting): handle_auth_done global_id 74529 payload 824 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 -- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 msgr2=0x7f9c381203a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:53.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 -- 172.21.15.5:0/2655367012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3809f780 msgr2=0x7f9c38136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3809f780 0x7f9c38136580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 monclient: found mon.noname-a 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 -- 172.21.15.5:0/2655367012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c280094b0 con 0x7f9c380a2bf0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c2f7fe700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c3811fe60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9c200011e0 tx=0x7f9c2000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 client.?.objecter ms_handle_connect 0x7f9c380a2bf0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20004ca0 con 0x7f9c380a2bf0 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:53.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: dump: 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: _finish_auth 0 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: _check_auth_tickets 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20004e00 con 0x7f9c380a2bf0 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 5 monclient: authenticate success, global_id 74529 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _renew_subs 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38123450 con 0x7f9c380a2bf0 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20007310 con 0x7f9c380a2bf0 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c357fa700 10 monclient: dump: 2024-09-16T11:05:53.229 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 client.74529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _renew_subs 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c38121920 con 0x7f9c380a2bf0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 1 librados: init done 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.204+0000 7f9c3d292700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c3d292700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c3d292700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c380a2fd0 con 0x7f9c380a2bf0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f9c20007470 con 0x7f9c380a2bf0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 1 --2- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 0x7f9c18039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f9c20047d00 con 0x7f9c380a2bf0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 10 client.74529.objecter ms_dispatch 0x7f9c38000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 3 client.74529.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 3 client.74529.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 20 client.74529.objecter dump_active .. 0 homeless 2024-09-16T11:05:53.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c37fff700 1 --2- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 0x7f9c18039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c37fff700 10 monclient: get_auth_request con 0x7f9c18036f60 auth_method 0 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c357fa700 10 client.74529.objecter ms_handle_connect 0x7f9c18036f60 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.208+0000 7f9c37fff700 1 --2- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 0x7f9c18039420 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7f9c280090b0 tx=0x7f9c28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.212+0000 7f9c36ffd700 4 set_mon_vals no callback set 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.216+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9c20007720 con 0x7f9c380a2bf0 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.216+0000 7f9c357fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:53.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.216+0000 7f9c357fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c3d292700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:05:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c3d292700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c3d292700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9c38123970 con 0x7f9c380a2bf0 2024-09-16T11:05:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c357fa700 1 -- 172.21.15.5:0/2655367012 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1531 (secure 0 0 0) 0x7f9c20011360 con 0x7f9c380a2bf0 2024-09-16T11:05:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c357fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.444+0000 7f9c357fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:53.454 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":101,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:05:53.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 10 librados: watch_flush enter 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 10 librados: watch_flush exit 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 20 client.74529.objecter shutdown clearing up homeless session... 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 10 client.74529.objecter successfully canceled tick 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 msgr2=0x7f9c18039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 0x7f9c18039420 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7f9c280090b0 tx=0x7f9c28008030 comp rx=0 tx=0).stop 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 10 monclient: shutdown 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 msgr2=0x7f9c3811fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c3811fe60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9c200011e0 tx=0x7f9c2000c800 comp rx=0 tx=0).stop 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 shutdown_connections 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f9c380a3510 0x7f9c381203a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9c18036f60 0x7f9c18039420 unknown :-1 s=CLOSED pgs=3890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c380a2bf0 0x7f9c3811fe60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 --2- 172.21.15.5:0/2655367012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c3809f780 0x7f9c38136580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 >> 172.21.15.5:0/2655367012 conn(0x7f9c380a53e0 msgr2=0x7f9c381bdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 shutdown_connections 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 -- 172.21.15.5:0/2655367012 wait complete. 2024-09-16T11:05:53.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:53.448+0000 7f9c3d292700 1 librados: shutdown 2024-09-16T11:05:56.472 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: get_monmap_and_config 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: build_initial_monmap 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: monmap: 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.592 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:56.590753+0000 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:56.590753+0000 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:56.593 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 Processor -- start 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 -- start start 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: init 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f7df4071870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7df40a10f0 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:56.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7df40a1510 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df409caa0 con 0x7f7df40a1510 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df409e3a0 con 0x7f7df40a10f0 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df409f700 con 0x7f7df4071870 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df3fff700 10 monclient(hunting): authenticate will time out at 1815.167207s 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df37fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7df40a1510 0x7f7df409eff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33026/0 (socket says 172.21.15.5:33026) 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df37fe700 1 -- 172.21.15.5:0/746639865 learned_addr learned my addr 172.21.15.5:0/746639865 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37188/0 (socket says 172.21.15.5:37188) 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request con 0x7f7df4071870 auth_method 0 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 msgr2=0x7f7df409aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 msgr2=0x7f7df409aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.588+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632725736 0 0) 0x7f7dec001490 con 0x7f7df40a1510 2024-09-16T11:05:56.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _negotiate 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): my global_id is 74532 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ddc0035d0 con 0x7f7df40a1510 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987000130 0 0) 0x7f7df409e3a0 con 0x7f7df40a10f0 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _negotiate 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): my global_id is 94216 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec001490 con 0x7f7df40a10f0 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150200119 0 0) 0x7f7ddc0035d0 con 0x7f7df40a1510 2024-09-16T11:05:56.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 msgr2=0x7f7df409aa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7df409aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 >> v1:172.21.15.143:6789/0 conn(0x7f7df40a10f0 legacy=0x7f7df409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _finish_hunting 0 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 monclient: found mon.noname-a-legacy 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df409b5f0 con 0x7f7df40a1510 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _finish_auth 0 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 5 monclient: authenticate success, global_id 74532 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:56.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f7dec001ba0 con 0x7f7df40a1510 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: dump: 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.611 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 >> v1:172.21.15.5:6789/0 conn(0x7f7df40a1510 legacy=0x7f7df409eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ddc003be0 0x7f7ddc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): picked mon.a con 0x7f7ddc003be0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7ddc004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): picked mon.c con 0x7f7df4071870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:56.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ddc004e20 0x7f7ddc0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): picked mon.b con 0x7f7ddc004e20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df409f700 con 0x7f7ddc003be0 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc00dee0 con 0x7f7ddc004e20 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc00e0a0 con 0x7f7df4071870 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ddc003be0 0x7f7ddc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request con 0x7f7ddc003be0 auth_method 0 2024-09-16T11:05:56.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7ddc004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): get_auth_request con 0x7f7df4071870 auth_method 0 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ddc004e20 0x7f7ddc0052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 10 monclient(hunting): get_auth_request con 0x7f7ddc004e20 auth_method 0 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient(hunting): handle_auth_done global_id 74532 payload 293 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ddc004e20 msgr2=0x7f7ddc0052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ddc004e20 0x7f7ddc0052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ddc003be0 msgr2=0x7f7ddc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ddc003be0 0x7f7ddc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient: _finish_hunting 0 2024-09-16T11:05:56.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 monclient: found mon.c 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 -- 172.21.15.5:0/746639865 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc0035d0 con 0x7f7df4071870 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient: _finish_auth 0 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 10 monclient: _check_auth_tickets 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df27fc700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7ddc004870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001960 tx=0x7f7ddc0058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ad285645b1e4fe6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4001450 con 0x7f7df4071870 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: dump: 2024-09-16T11:05:56.616 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:56.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de40052f0 con 0x7f7df4071870 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 4 set_mon_vals no callback set 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 1 -- 172.21.15.5:0/746639865 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40036a0 con 0x7f7df4071870 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df1ffb700 10 monclient: dump: 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.618 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 10 monclient: shutdown 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 -- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 msgr2=0x7f7ddc004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 --2- 172.21.15.5:0/746639865 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df4071870 0x7f7ddc004870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001960 tx=0x7f7ddc0058a0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df37fe700 1 -- 172.21.15.5:0/746639865 reap_dead start 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 -- 172.21.15.5:0/746639865 shutdown_connections 2024-09-16T11:05:56.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 -- 172.21.15.5:0/746639865 >> 172.21.15.5:0/746639865 conn(0x7f7df40a45d0 msgr2=0x7f7df40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 -- 172.21.15.5:0/746639865 shutdown_connections 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 1 -- 172.21.15.5:0/746639865 wait complete. 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 10 monclient: build_initial_monmap 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.592+0000 7f7df3fff700 10 monclient: monmap: 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:56.620 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 librados: starting msgr at 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 librados: starting objecter 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 Processor -- start 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- start start 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 librados: setting wanted keys 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 librados: calling monclient init 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: init 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:05:56.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 0x7f7df41c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f7df4071870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f7df41c0560 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f7df41c09e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41a7620 con 0x7f7df41c09e0 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40707e0 con 0x7f7df4071870 2024-09-16T11:05:56.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df409b0d0 con 0x7f7df41c0560 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient(hunting): authenticate will time out at 1815.172833s 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37214/0 (socket says 172.21.15.5:37214) 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52760/0 (socket says 172.21.15.5:52760) 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 -- 172.21.15.5:0/2578973008 learned_addr learned my addr 172.21.15.5:0/2578973008 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 10 monclient(hunting): get_auth_request con 0x7f7df41c0560 auth_method 0 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): get_auth_request con 0x7f7df41c09e0 auth_method 0 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 0x7f7df41c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request con 0x7f7df4071870 auth_method 0 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 -- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 msgr2=0x7f7df41c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 -- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 msgr2=0x7f7df41c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:56.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df27fc700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:56.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient(hunting): handle_auth_done global_id 94219 payload 824 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 -- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 msgr2=0x7f7df41c6510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 -- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 msgr2=0x7f7df41c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 monclient: found mon.noname-b 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 -- 172.21.15.5:0/2578973008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df41a5430 con 0x7f7df4071870 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df37fe700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:56.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df2ffd700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 0x7f7df41c2fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7de8015aa0 tx=0x7f7de8015db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 client.?.objecter ms_handle_connect 0x7f7df4071870 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8009620 con 0x7f7df4071870 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: dump: 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.627 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: _finish_auth 0 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 5 monclient: authenticate success, global_id 94219 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _renew_subs 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df40a3630 con 0x7f7df4071870 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de8016040 con 0x7f7df4071870 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:56.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de8009980 con 0x7f7df4071870 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 client.94219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7ddb7fe700 10 monclient: dump: 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.629 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _renew_subs 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df17fa700 4 set_mon_vals no callback set 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df41a5b80 con 0x7f7df4071870 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.596+0000 7f7df3fff700 1 librados: init done 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df3fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df409b3e0 con 0x7f7df4071870 2024-09-16T11:05:56.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f7de8009ae0 con 0x7f7df4071870 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 1 --2- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 0x7f7de0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f7de804c2f0 con 0x7f7df4071870 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 10 client.94219.objecter ms_dispatch 0x7f7df4000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 3 client.94219.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 3 client.94219.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 20 client.94219.objecter dump_active .. 0 homeless 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df27fc700 1 --2- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 0x7f7de0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df27fc700 10 monclient: get_auth_request con 0x7f7de0036f60 auth_method 0 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7ddb7fe700 10 client.94219.objecter ms_handle_connect 0x7f7de0036f60 2024-09-16T11:05:56.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.600+0000 7f7df27fc700 1 --2- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 0x7f7de0039420 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001120 tx=0x7f7de400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:56.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.612+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7de8023770 con 0x7f7df4071870 2024-09-16T11:05:56.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.612+0000 7f7ddb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:56.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.612+0000 7f7ddb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7df3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:05:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7df3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7df3fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:05:56.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7df409b870 con 0x7f7df4071870 2024-09-16T11:05:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7ddb7fe700 1 -- 172.21.15.5:0/2578973008 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f7de801a3e0 con 0x7f7df4071870 2024-09-16T11:05:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7ddb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:05:56.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.848+0000 7f7ddb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:56.853 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:05:56.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 10 librados: watch_flush enter 2024-09-16T11:05:56.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 10 librados: watch_flush exit 2024-09-16T11:05:56.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 20 client.94219.objecter shutdown clearing up homeless session... 2024-09-16T11:05:56.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 10 client.94219.objecter successfully canceled tick 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 msgr2=0x7f7de0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 0x7f7de0039420 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001120 tx=0x7f7de400a000 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 10 monclient: shutdown 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 msgr2=0x7f7df41c2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 0x7f7df41c2fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7de8015aa0 tx=0x7f7de8015db0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 shutdown_connections 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> v2:172.21.15.143:6812/1098205034 conn(0x7f7de0036f60 0x7f7de0039420 unknown :-1 s=CLOSED pgs=3892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7df41c09e0 0x7f7df41c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f7df41c0560 0x7f7df41c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 --2- 172.21.15.5:0/2578973008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df4071870 0x7f7df41c2fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 >> 172.21.15.5:0/2578973008 conn(0x7f7df40a45d0 msgr2=0x7f7df406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:56.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 shutdown_connections 2024-09-16T11:05:56.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 -- 172.21.15.5:0/2578973008 wait complete. 2024-09-16T11:05:56.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.852+0000 7f7df3fff700 1 librados: shutdown 2024-09-16T11:05:56.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:05:56.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:05:56.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: get_monmap_and_config 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: build_initial_monmap 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: monmap: 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:56.979328+0000 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:56.979328+0000 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.981 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 Processor -- start 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 -- start start 2024-09-16T11:05:56.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: init 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5d18070750 0x7f5d18070b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-a con 0x7f5d18070750 addr v2:172.21.15.5:3300/0 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d180a04b0 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d180a14c0 0x7f5d18071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-c con 0x7f5d180a14c0 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1809fb60 con 0x7f5d180a04b0 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d180715b0 con 0x7f5d18070750 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:56.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d180a0090 con 0x7f5d180a14c0 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1ed2a700 10 monclient(hunting): authenticate will time out at 1815.555727s 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.976+0000 7f5d1dd28700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5d18070750 0x7f5d18070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5d18070750 0x7f5d18070b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52770/0 (socket says 172.21.15.5:52770) 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 -- 172.21.15.5:0/1761653916 learned_addr learned my addr 172.21.15.5:0/1761653916 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): get_auth_request con 0x7f5d18070750 auth_method 0 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 1 --2- 172.21.15.5:0/1761653916 >> v2:172.21.15.143:3300/0 conn(0x7f5d180a14c0 0x7f5d18071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): get_auth_request con 0x7f5d180a14c0 auth_method 0 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:56.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:56.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:56.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1e529700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_done global_id 74538 payload 274 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 -- 172.21.15.5:0/1761653916 >> v2:172.21.15.143:3300/0 conn(0x7f5d180a14c0 msgr2=0x7f5d18071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 --2- 172.21.15.5:0/1761653916 >> v2:172.21.15.143:3300/0 conn(0x7f5d180a14c0 0x7f5d18071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 -- 172.21.15.5:0/1761653916 >> v1:172.21.15.143:6789/0 conn(0x7f5d180a04b0 legacy=0x7f5d181326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient: _finish_hunting 0 2024-09-16T11:05:56.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 monclient: found mon.noname-a 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 -- 172.21.15.5:0/1761653916 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1809f7f0 con 0x7f5d18070750 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1dd28700 1 --2- 172.21.15.5:0/1761653916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d18070b30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d1809ef80 tx=0x7f5d08009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7d07ac95142e26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 1 -- 172.21.15.5:0/1761653916 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08011830 con 0x7f5d18070750 2024-09-16T11:05:56.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: dump: 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.993 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: _finish_auth 0 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: _check_auth_tickets 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 1 -- 172.21.15.5:0/1761653916 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08011990 con 0x7f5d18070750 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 5 monclient: authenticate success, global_id 74538 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:56.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 4 set_mon_vals no callback set 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 1 -- 172.21.15.5:0/1761653916 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08011d20 con 0x7f5d18070750 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1cd26700 10 monclient: dump: 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:56.995 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 10 monclient: shutdown 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/1761653916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 msgr2=0x7f5d18070b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/1761653916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d18070b30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d1809ef80 tx=0x7f5d08009bf0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/1761653916 shutdown_connections 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/1761653916 >> v2:172.21.15.143:3300/0 conn(0x7f5d180a14c0 0x7f5d18071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/1761653916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d18070b30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/1761653916 >> 172.21.15.5:0/1761653916 conn(0x7f5d180a45d0 msgr2=0x7f5d180a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/1761653916 shutdown_connections 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/1761653916 wait complete. 2024-09-16T11:05:56.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 10 monclient: build_initial_monmap 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 10 monclient: monmap: 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:56.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 librados: starting msgr at 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 librados: starting objecter 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 Processor -- start 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 -- start start 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 librados: setting wanted keys 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 1 librados: calling monclient init 2024-09-16T11:05:56.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.980+0000 7f5d1ed2a700 10 monclient: init 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d1811de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-a con 0x7f5d18070750 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d180a04b0 0x7f5d1811e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-b con 0x7f5d180a04b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 0x7f5d1811e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): picked mon.noname-c con 0x7f5d180a14c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:57.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18121ba0 con 0x7f5d18070750 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18120410 con 0x7f5d180a04b0 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18132740 con 0x7f5d180a14c0 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): authenticate will time out at 1815.560901s 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d1811de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d1811de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52772/0 (socket says 172.21.15.5:52772) 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 -- 172.21.15.5:0/2757608148 learned_addr learned my addr 172.21.15.5:0/2757608148 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): get_auth_request con 0x7f5d18070750 auth_method 0 2024-09-16T11:05:57.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 0x7f5d1811e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 10 monclient(hunting): get_auth_request con 0x7f5d180a14c0 auth_method 0 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d180a04b0 0x7f5d1811e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 msgr2=0x7f5d1811e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 msgr2=0x7f5d1811e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 0x7f5d1811e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): get_auth_request con 0x7f5d180a04b0 auth_method 0 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1e529700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 0x7f5d1811e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient(hunting): handle_auth_done global_id 74541 payload 824 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 msgr2=0x7f5d1811e890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f5d180a14c0 0x7f5d1811e890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d180a04b0 msgr2=0x7f5d1811e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d180a04b0 0x7f5d1811e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient: _finish_hunting 0 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 monclient: found mon.noname-a 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d08009570 con 0x7f5d18070750 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1dd28700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d1811de10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d18075050 tx=0x7f5d08004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 client.?.objecter ms_handle_connect 0x7f5d18070750 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08004340 con 0x7f5d18070750 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: dump: 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: _finish_auth 0 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 5 monclient: authenticate success, global_id 74541 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _renew_subs 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0801f070 con 0x7f5d18070750 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1809cad0 con 0x7f5d18070750 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d080036c0 con 0x7f5d18070750 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0ffff700 4 set_mon_vals no callback set 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 client.74541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 10 monclient: dump: 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _renew_subs 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d181218c0 con 0x7f5d18070750 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 librados: init done 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 0x7f5d1811a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d18067380 con 0x7f5d1811a530 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f5d08021030 con 0x7f5d18070750 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 0x7f5d1811a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient: get_auth_request con 0x7f5d1811a530 auth_method 0 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.984+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1d527700 10 monclient(hunting): handle_auth_done global_id 74544 payload 824 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1d527700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d18132dc0 con 0x7f5d1811a530 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 --2- 172.21.15.5:0/2757608148 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5cfc036f60 0x7f5cfc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f5d080490d0 con 0x7f5d18070750 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1d527700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 0x7f5d1811a910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5d18120130 tx=0x7f5d1400fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 client.74541.objecter ms_dispatch 0x7f5d18000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 3 client.74541.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 3 client.74541.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 20 client.74541.objecter dump_active .. 0 homeless 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 client.74541.objecter ms_handle_connect 0x7f5d1811a530 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 client.74541.objecter resend_mon_ops 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d140052e0 con 0x7f5d1811a530 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 monclient: dump: 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1e529700 1 --2- 172.21.15.5:0/2757608148 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5cfc036f60 0x7f5cfc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1e529700 10 monclient: get_auth_request con 0x7f5cfc036f60 auth_method 0 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f5d14005440 con 0x7f5d1811a530 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 msgr2=0x7f5d1811a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 0x7f5d1811a910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5d18120130 tx=0x7f5d1400fb00 comp rx=0 tx=0).stop 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 msgr2=0x7f5d1811a910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:57.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811a530 0x7f5d1811a910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d1e529700 1 --2- 172.21.15.5:0/2757608148 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5cfc036f60 0x7f5cfc039420 secure :-1 s=READY pgs=3893 cs=0 l=1 rev1=1 crypto rx=0x7f5d180a18d0 tx=0x7f5d1000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:56.988+0000 7f5d0e7fc700 10 client.74541.objecter ms_handle_connect 0x7f5cfc036f60 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 0x7f5d181d4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d181331c0 con 0x7f5d1811ae50 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 0x7f5d181d4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient: get_auth_request con 0x7f5d1811ae50 auth_method 0 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 10 monclient(hunting): handle_auth_done global_id 74547 payload 824 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 1 -- 172.21.15.5:0/2757608148 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5d180008d0 con 0x7f5d1811ae50 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 client.74541.objecter ms_handle_connect 0x7f5d1811ae50 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1d527700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 0x7f5d181d4bb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5d140015a0 tx=0x7f5d14001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 client.74541.objecter resend_mon_ops 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14023070 con 0x7f5d1811ae50 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 monclient: dump: 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.021 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4779 (secure 0 0 0) 0x7f5d14006880 con 0x7f5d1811ae50 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:57.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 msgr2=0x7f5d181d4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 0x7f5d181d4bb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5d140015a0 tx=0x7f5d14001120 comp rx=0 tx=0).stop 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 msgr2=0x7f5d181d4bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d0e7fc700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d1811ae50 0x7f5d181d4bb0 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5d140015a0 tx=0x7f5d14001120 comp rx=0 tx=0).stop 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:05:57.023 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 3, 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:57.024 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:57.025 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:05:57.026 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:05:57.027 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.028 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.029 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.030 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:57.031 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:05:57.032 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.033 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:57.034 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.035 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:57.036 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:05:57.037 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:05:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 librados: watch_flush enter 2024-09-16T11:05:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 librados: watch_flush exit 2024-09-16T11:05:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 20 client.74541.objecter shutdown clearing up homeless session... 2024-09-16T11:05:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 client.74541.objecter successfully canceled tick 2024-09-16T11:05:57.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5cfc036f60 msgr2=0x7f5cfc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/2757608148 >> v2:172.21.15.143:6812/1098205034 conn(0x7f5cfc036f60 0x7f5cfc039420 secure :-1 s=READY pgs=3893 cs=0 l=1 rev1=1 crypto rx=0x7f5d180a18d0 tx=0x7f5d1000b040 comp rx=0 tx=0).stop 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 10 monclient: shutdown 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 msgr2=0x7f5d1811de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 --2- 172.21.15.5:0/2757608148 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5d18070750 0x7f5d1811de10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d18075050 tx=0x7f5d08004770 comp rx=0 tx=0).stop 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1e529700 1 -- 172.21.15.5:0/2757608148 reap_dead start 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1e529700 1 -- 172.21.15.5:0/2757608148 reap_dead start 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 shutdown_connections 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 >> 172.21.15.5:0/2757608148 conn(0x7f5d180a45d0 msgr2=0x7f5d18131f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 shutdown_connections 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 -- 172.21.15.5:0/2757608148 wait complete. 2024-09-16T11:05:57.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.000+0000 7f5d1ed2a700 1 librados: shutdown 2024-09-16T11:05:57.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:05:57.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:05:57.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: get_monmap_and_config 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: build_initial_monmap 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: monmap: 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:57.161535+0000 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:57.161535+0000 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.163 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:57.164 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:57.164 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 Processor -- start 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 -- start start 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: init 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbec09e8e0 0x7fbbec09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-a con 0x7fbbec09e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:05:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 0x7fbbec139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-c con 0x7fbbec09f200 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-b con 0x7fbbec13a460 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbec09c030 con 0x7fbbec09e8e0 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbec13adc0 con 0x7fbbec09f200 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec09fb80 con 0x7fbbec13a460 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient(hunting): authenticate will time out at 1815.737892s 2024-09-16T11:05:57.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbec09e8e0 0x7fbbec09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf28eb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbec09e8e0 0x7fbbec09ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52816/0 (socket says 172.21.15.5:52816) 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 -- 172.21.15.5:0/1497145784 learned_addr learned my addr 172.21.15.5:0/1497145784 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request con 0x7fbbec09e8e0 auth_method 0 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/1497145784 >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 0x7fbbec139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf28eb700 1 -- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 msgr2=0x7fbbec13a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:05:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf28eb700 1 -- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 msgr2=0x7fbbec13a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:05:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf28eb700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): get_auth_request con 0x7fbbec09f200 auth_method 0 2024-09-16T11:05:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf28eb700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_done global_id 74550 payload 274 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 -- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 msgr2=0x7fbbec13a880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:05:57.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 -- 172.21.15.5:0/1497145784 >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 msgr2=0x7fbbec139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/1497145784 >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 0x7fbbec139f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient: _finish_hunting 0 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 monclient: found mon.noname-a 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 -- 172.21.15.5:0/1497145784 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec09b970 con 0x7fbbec09e8e0 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/1497145784 >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 0x7fbbec139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec09e8e0 0x7fbbec09ecc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1a70 tx=0x7fbbdc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaeaa410f9cadc6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 1 -- 172.21.15.5:0/1497145784 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc011830 con 0x7fbbec09e8e0 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: dump: 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.174 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: _finish_auth 0 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: _check_auth_tickets 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 1 -- 172.21.15.5:0/1497145784 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbdc011990 con 0x7fbbec09e8e0 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 5 monclient: authenticate success, global_id 74550 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 1 -- 172.21.15.5:0/1497145784 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc011d20 con 0x7fbbec09e8e0 2024-09-16T11:05:57.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf30ec700 4 set_mon_vals no callback set 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.160+0000 7fbbf10e8700 10 monclient: dump: 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.176 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 10 monclient: shutdown 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec09e8e0 msgr2=0x7fbbec09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec09e8e0 0x7fbbec09ecc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1a70 tx=0x7fbbdc009bf0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- 172.21.15.5:0/1497145784 shutdown_connections 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec13a460 0x7fbbec13a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/1497145784 >> v2:172.21.15.143:3300/0 conn(0x7fbbec09f200 0x7fbbec139f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/1497145784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec09e8e0 0x7fbbec09ecc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- 172.21.15.5:0/1497145784 >> 172.21.15.5:0/1497145784 conn(0x7fbbec0a45d0 msgr2=0x7fbbec0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- 172.21.15.5:0/1497145784 shutdown_connections 2024-09-16T11:05:57.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- 172.21.15.5:0/1497145784 wait complete. 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 10 monclient: build_initial_monmap 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 10 monclient: monmap: 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:57.185 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 librados: starting msgr at 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 librados: starting objecter 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 Processor -- start 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 -- start start 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 librados: setting wanted keys 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 1 librados: calling monclient init 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.164+0000 7fbbf30ec700 10 monclient: init 2024-09-16T11:05:57.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbec09e8e0 0x7fbbec1ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-b con 0x7fbbec09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-c con 0x7fbbec09f200 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 0x7fbbec139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): picked mon.noname-a con 0x7fbbec13a460 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec1ceb20 con 0x7fbbec13a460 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec126900 con 0x7fbbec09e8e0 2024-09-16T11:05:57.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec126be0 con 0x7fbbec09f200 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient(hunting): authenticate will time out at 1815.745081s 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 0x7fbbec139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37244/0 (socket says 172.21.15.5:37244) 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 -- 172.21.15.5:0/2010479350 learned_addr learned my addr 172.21.15.5:0/2010479350 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 10 monclient(hunting): get_auth_request con 0x7fbbec09f200 auth_method 0 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): get_auth_request con 0x7fbbec13a460 auth_method 0 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 msgr2=0x7fbbec139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 msgr2=0x7fbbec139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbec09e8e0 0x7fbbec1ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request con 0x7fbbec09e8e0 auth_method 0 2024-09-16T11:05:57.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient(hunting): handle_auth_done global_id 74553 payload 824 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 msgr2=0x7fbbec139200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:05:57.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec09f200 0x7fbbec139200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbec09e8e0 msgr2=0x7fbbec1ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbec09e8e0 0x7fbbec1ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient: _finish_hunting 0 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 monclient: found mon.noname-a 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80094b0 con 0x7fbbec13a460 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbec09e8e0 0x7fbbec1ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf28eb700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 0x7fbbec139740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400dc60 tx=0x7fbbe400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 client.?.objecter ms_handle_connect 0x7fbbec13a460 2024-09-16T11:05:57.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4003c40 con 0x7fbbec13a460 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: dump: 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.192 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: _finish_auth 0 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:05:57.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 5 monclient: authenticate success, global_id 74553 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4003dc0 con 0x7fbbec13a460 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe400ba90 con 0x7fbbec13a460 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _renew_subs 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbdc009570 con 0x7fbbec13a460 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf08e7700 4 set_mon_vals no callback set 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:57.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbe2ffd700 10 monclient: dump: 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:57.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 client.74553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _renew_subs 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec09b0d0 con 0x7fbbec13a460 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.168+0000 7fbbf30ec700 1 librados: init done 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fbbe400e040 con 0x7fbbec13a460 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf30ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec067380 con 0x7fbbec124850 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 10 monclient: get_auth_request con 0x7fbbec124850 auth_method 0 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 1 --2- 172.21.15.5:0/2010479350 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbbd0036f60 0x7fbbd0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:05:57.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7fbbe4047bf0 con 0x7fbbec13a460 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 10 client.74553.objecter ms_dispatch 0x7fbbec000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 3 client.74553.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 3 client.74553.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 20 client.74553.objecter dump_active .. 0 homeless 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/2010479350 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbbd0036f60 0x7fbbd0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf18e9700 10 monclient: get_auth_request con 0x7fbbd0036f60 auth_method 0 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbf18e9700 1 --2- 172.21.15.5:0/2010479350 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbbd0036f60 0x7fbbd0039420 secure :-1 s=READY pgs=3894 cs=0 l=1 rev1=1 crypto rx=0x7fbbec126270 tx=0x7fbbe8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:57.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.172+0000 7fbbe2ffd700 10 client.74553.objecter ms_handle_connect 0x7fbbd0036f60 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 10 monclient: get_auth_request con 0x7fbbec124850 auth_method 0 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.376+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:05:57.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 10 monclient: get_auth_request con 0x7fbbec124850 auth_method 0 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:05:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:57.780+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:05:58.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient: get_auth_request con 0x7fbbec124850 auth_method 0 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_done global_id 104099 payload 824 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbec09b3e0 con 0x7fbbec124850 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 client.74553.objecter ms_handle_connect 0x7fbbec124850 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 client.74553.objecter resend_mon_ops 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1a70 tx=0x7fbbdc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc020050 con 0x7fbbec124850 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 monclient: dump: 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.589 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fbbdc021040 con 0x7fbbec124850 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1a70 tx=0x7fbbdc004770 comp rx=0 tx=0).stop 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 msgr2=0x7fbbec122f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:58.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.584+0000 7fbbe2ffd700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec124850 0x7fbbec122f80 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbbec0a1a70 tx=0x7fbbdc004770 comp rx=0 tx=0).stop 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf30ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 0x7fbbec1234c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf30ec700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec09bce0 con 0x7fbbec126340 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 0x7fbbec1234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient: get_auth_request con 0x7fbbec126340 auth_method 0 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.596+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf20ea700 10 monclient(hunting): handle_auth_done global_id 104102 payload 824 2024-09-16T11:05:58.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf20ea700 1 -- 172.21.15.5:0/2010479350 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbbec09f610 con 0x7fbbec126340 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf20ea700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 0x7fbbec1234c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009150 tx=0x7fbbdc009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 client.74553.objecter ms_handle_connect 0x7fbbec126340 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 client.74553.objecter resend_mon_ops 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc030be0 con 0x7fbbec126340 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 monclient: dump: 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.605 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7fbbdc034070 con 0x7fbbec126340 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 msgr2=0x7fbbec1234c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 0x7fbbec1234c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009150 tx=0x7fbbdc009bf0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 msgr2=0x7fbbec1234c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbe2ffd700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fbbec126340 0x7fbbec1234c0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009150 tx=0x7fbbdc009bf0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:05:58.607 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 5, 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.608 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.609 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:05:58.610 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.611 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.612 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.613 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.614 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:05:58.615 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.616 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:58.617 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:05:58.618 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:05:58.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 10 librados: watch_flush enter 2024-09-16T11:05:58.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 10 librados: watch_flush exit 2024-09-16T11:05:58.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 20 client.74553.objecter shutdown clearing up homeless session... 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 10 client.74553.objecter successfully canceled tick 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbbd0036f60 msgr2=0x7fbbd0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/2010479350 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbbd0036f60 0x7fbbd0039420 secure :-1 s=READY pgs=3894 cs=0 l=1 rev1=1 crypto rx=0x7fbbec126270 tx=0x7fbbe8008040 comp rx=0 tx=0).stop 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 10 monclient: shutdown 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf28eb700 1 -- 172.21.15.5:0/2010479350 reap_dead start 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 msgr2=0x7fbbec139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 0x7fbbec139740 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400dc60 tx=0x7fbbe400df70 comp rx=0 tx=0).stop 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 shutdown_connections 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 --2- 172.21.15.5:0/2010479350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbec13a460 0x7fbbec139740 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 >> 172.21.15.5:0/2010479350 conn(0x7fbbec0a45d0 msgr2=0x7fbbec1c2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 shutdown_connections 2024-09-16T11:05:58.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 -- 172.21.15.5:0/2010479350 wait complete. 2024-09-16T11:05:58.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.600+0000 7fbbf30ec700 1 librados: shutdown 2024-09-16T11:05:58.622 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:05:58.622 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:05:58.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:05:58.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: get_monmap_and_config 2024-09-16T11:05:58.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: build_initial_monmap 2024-09-16T11:05:58.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: monmap: 2024-09-16T11:05:58.747 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:58.746121+0000 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:58.746121+0000 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:58.748 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 Processor -- start 2024-09-16T11:05:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 -- start start 2024-09-16T11:05:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: init 2024-09-16T11:05:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:58.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6db00a0db0 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6db00a18a0 0x7f6db006dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-c con 0x7f6db00a18a0 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6db013b2a0 addr v1:172.21.15.143:6789/0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00a24f0 con 0x7f6db00a0db0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db006c790 con 0x7f6db013b2a0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db00a2b40 con 0x7f6db00a18a0 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6db4f8a700 10 monclient(hunting): authenticate will time out at 1817.323393s 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dae59c700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6db00a0db0 0x7f6db00a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33034/0 (socket says 172.21.15.5:33034) 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dae59c700 1 -- 172.21.15.5:0/3742206071 learned_addr learned my addr 172.21.15.5:0/3742206071 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:58.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dadd9b700 1 --2- 172.21.15.5:0/3742206071 >> v2:172.21.15.143:3300/0 conn(0x7f6db00a18a0 0x7f6db006dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dadd9b700 10 monclient(hunting): get_auth_request con 0x7f6db00a18a0 auth_method 0 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dadd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dadd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.744+0000 7f6dadd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_done global_id 94237 payload 274 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 1 -- 172.21.15.5:0/3742206071 >> v1:172.21.15.143:6789/0 conn(0x7f6db013b2a0 legacy=0x7f6db013d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 1 -- 172.21.15.5:0/3742206071 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422632770 0 0) 0x7f6db00a24f0 con 0x7f6db00a0db0 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 1 -- 172.21.15.5:0/3742206071 >> v1:172.21.15.5:6789/0 conn(0x7f6db00a0db0 legacy=0x7f6db00a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient: _finish_hunting 0 2024-09-16T11:05:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 1 monclient: found mon.noname-c 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 1 -- 172.21.15.5:0/3742206071 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00a26c0 con 0x7f6db00a18a0 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dadd9b700 1 --2- 172.21.15.5:0/3742206071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db006dca0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009850 tx=0x7f6da0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21bf67cbb371c93d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 1 -- 172.21.15.5:0/3742206071 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0011830 con 0x7f6db00a18a0 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: dump: 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: _finish_auth 0 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: _check_auth_tickets 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 5 monclient: authenticate success, global_id 94237 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 1 -- 172.21.15.5:0/3742206071 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da0011990 con 0x7f6db00a18a0 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:58.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 4 set_mon_vals no callback set 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 1 -- 172.21.15.5:0/3742206071 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0011d20 con 0x7f6db00a18a0 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6dad59a700 10 monclient: dump: 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.757 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 10 monclient: shutdown 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- 172.21.15.5:0/3742206071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 msgr2=0x7f6db006dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/3742206071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db006dca0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009850 tx=0x7f6da0009b60 comp rx=0 tx=0).stop 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- 172.21.15.5:0/3742206071 shutdown_connections 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/3742206071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db006dca0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- 172.21.15.5:0/3742206071 >> 172.21.15.5:0/3742206071 conn(0x7f6db00a64b0 msgr2=0x7f6db00a68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- 172.21.15.5:0/3742206071 shutdown_connections 2024-09-16T11:05:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- 172.21.15.5:0/3742206071 wait complete. 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 10 monclient: build_initial_monmap 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 10 monclient: monmap: 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:58.764 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 librados: starting msgr at 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 librados: starting objecter 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 Processor -- start 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 -- start start 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 librados: setting wanted keys 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 1 librados: calling monclient init 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.748+0000 7f6db4f8a700 10 monclient: init 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6db00a0db0 0x7f6db01caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-a con 0x7f6db00a0db0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db01cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-b con 0x7f6db00a18a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 0x7f6db01cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): picked mon.noname-c con 0x7f6db013b2a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db01395c0 con 0x7f6db00a0db0 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db009d6a0 con 0x7f6db00a18a0 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00a4540 con 0x7f6db013b2a0 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient(hunting): authenticate will time out at 1817.328867s 2024-09-16T11:05:58.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6db00a0db0 0x7f6db01caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 0x7f6db01cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 0x7f6db01cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37314/0 (socket says 172.21.15.5:37314) 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 learned_addr learned my addr 172.21.15.5:0/1426110734 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): get_auth_request con 0x7f6db013b2a0 auth_method 0 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): get_auth_request con 0x7f6db00a0db0 auth_method 0 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db01cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): get_auth_request con 0x7f6db00a18a0 auth_method 0 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6dadd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient(hunting): handle_auth_done global_id 104105 payload 824 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 msgr2=0x7f6db01cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db00a18a0 0x7f6db01cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6db00a0db0 msgr2=0x7f6db01caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6db00a0db0 0x7f6db01caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient: _finish_hunting 0 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 monclient: found mon.noname-c 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da00094b0 con 0x7f6db013b2a0 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6daed9d700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 0x7f6db01cb720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6da800cbf0 tx=0x7f6da800cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 client.?.objecter ms_handle_connect 0x7f6db013b2a0 2024-09-16T11:05:58.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8003c20 con 0x7f6db013b2a0 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: dump: 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.770 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: _finish_auth 0 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: _check_auth_tickets 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 5 monclient: authenticate success, global_id 104105 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _renew_subs 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8003d80 con 0x7f6db013b2a0 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db0137740 con 0x7f6db013b2a0 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da800aaa0 con 0x7f6db013b2a0 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:58.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 client.104105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9effd700 10 monclient: dump: 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _renew_subs 2024-09-16T11:05:58.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db0139aa0 con 0x7f6db013b2a0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 1 librados: init done 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6d9ffff700 4 set_mon_vals no callback set 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.752+0000 7f6db4f8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 0x7f6db009c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00677f0 con 0x7f6db009c4f0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6da800ac00 con 0x7f6db013b2a0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 0x7f6db009c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 --2- 172.21.15.5:0/1426110734 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d90036f60 0x7f6d90039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient: get_auth_request con 0x7f6db009c4f0 auth_method 0 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f6da8047f90 con 0x7f6db013b2a0 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 client.104105.objecter ms_dispatch 0x7f6db0000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 3 client.104105.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 3 client.104105.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 20 client.104105.objecter dump_active .. 0 homeless 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dadd9b700 1 --2- 172.21.15.5:0/1426110734 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d90036f60 0x7f6d90039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dadd9b700 10 monclient: get_auth_request con 0x7f6d90036f60 auth_method 0 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 10 monclient(hunting): handle_auth_done global_id 94246 payload 824 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6db00a3700 con 0x7f6db009c4f0 2024-09-16T11:05:58.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 client.104105.objecter ms_handle_connect 0x7f6db009c4f0 2024-09-16T11:05:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 client.104105.objecter resend_mon_ops 2024-09-16T11:05:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dae59c700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 0x7f6db009c8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f60 tx=0x7f6da4001910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 client.104105.objecter ms_handle_connect 0x7f6d90036f60 2024-09-16T11:05:58.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6dadd9b700 1 --2- 172.21.15.5:0/1426110734 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d90036f60 0x7f6d90039420 secure :-1 s=READY pgs=3895 cs=0 l=1 rev1=1 crypto rx=0x7f6da00090b0 tx=0x7f6da0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4017070 con 0x7f6db009c4f0 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 monclient: dump: 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.776 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f6da4012560 con 0x7f6db009c4f0 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 msgr2=0x7f6db009c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 0x7f6db009c8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f60 tx=0x7f6da4001910 comp rx=0 tx=0).stop 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 msgr2=0x7f6db009c8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:58.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.756+0000 7f6d9effd700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db009c4f0 0x7f6db009c8d0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f60 tx=0x7f6da4001910 comp rx=0 tx=0).stop 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.768+0000 7f6db4f8a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.768+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 0x7f6db002bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.768+0000 7f6db4f8a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.768+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db006d2c0 con 0x7f6db002b6a0 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 0x7f6db002bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient: get_auth_request con 0x7f6db002b6a0 auth_method 0 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 10 monclient(hunting): handle_auth_done global_id 94249 payload 824 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 1 -- 172.21.15.5:0/1426110734 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6db00a60f0 con 0x7f6db002b6a0 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 client.104105.objecter ms_handle_connect 0x7f6db002b6a0 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6dae59c700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 0x7f6db002bab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f60 tx=0x7f6da4000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 client.104105.objecter resend_mon_ops 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4023070 con 0x7f6db002b6a0 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 monclient: dump: 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.781 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f6da401e110 con 0x7f6db002b6a0 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 msgr2=0x7f6db002bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 0x7f6db002bab0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f60 tx=0x7f6da4000ed0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 msgr2=0x7f6db002bab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6d9effd700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6db002b6a0 0x7f6db002bab0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:05:58.782 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 5, 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:05:58.783 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:05:58.784 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:05:58.785 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:05:58.786 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:05:58.787 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.788 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:05:58.789 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.790 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:05:58.791 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:05:58.792 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:05:58.793 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:05:58.794 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 10 librados: watch_flush enter 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 10 librados: watch_flush exit 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 20 client.104105.objecter shutdown clearing up homeless session... 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 10 client.104105.objecter successfully canceled tick 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d90036f60 msgr2=0x7f6d90039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/1426110734 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d90036f60 0x7f6d90039420 secure :-1 s=READY pgs=3895 cs=0 l=1 rev1=1 crypto rx=0x7f6da00090b0 tx=0x7f6da0008030 comp rx=0 tx=0).stop 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 10 monclient: shutdown 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:58.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 msgr2=0x7f6db01cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 --2- 172.21.15.5:0/1426110734 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6db013b2a0 0x7f6db01cb720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6da800cbf0 tx=0x7f6da800cf00 comp rx=0 tx=0).stop 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 reap_dead start 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6daed9d700 1 -- 172.21.15.5:0/1426110734 reap_dead start 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 shutdown_connections 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 >> 172.21.15.5:0/1426110734 conn(0x7f6db00a64b0 msgr2=0x7f6db00a01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 shutdown_connections 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 -- 172.21.15.5:0/1426110734 wait complete. 2024-09-16T11:05:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.772+0000 7f6db4f8a700 1 librados: shutdown 2024-09-16T11:05:58.797 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:05:58.797 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T11:05:58.798 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T11:05:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 10 monclient: get_monmap_and_config 2024-09-16T11:05:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 10 monclient: build_initial_monmap 2024-09-16T11:05:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 10 monclient: monmap: 2024-09-16T11:05:58.915 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:05:58.914257+0000 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:05:58.914257+0000 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:05:58.916 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 1 Processor -- start 2024-09-16T11:05:58.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 1 -- start start 2024-09-16T11:05:58.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.912+0000 7fc01cbc5700 10 monclient: init 2024-09-16T11:05:58.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:58.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T11:05:58.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0180a36b0 0x7fc01806f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-a con 0x7fc0180a36b0 addr v2:172.21.15.5:3300/0 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 0x7fc0180a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-c con 0x7fc01806cc10 addr v2:172.21.15.143:3300/0 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0180a3fd0 addr v1:172.21.15.5:6789/0 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01806d090 con 0x7fc0180a3fd0 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01806e130 con 0x7fc0180a36b0 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01806e9d0 con 0x7fc01806cc10 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient(hunting): authenticate will time out at 1817.492348s 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0180a36b0 0x7fc01806f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc017fff700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc0180a3fd0 0x7fc018131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33044/0 (socket says 172.21.15.5:33044) 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc0180a36b0 0x7fc01806f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52852/0 (socket says 172.21.15.5:52852) 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc017fff700 1 -- 172.21.15.5:0/116493083 learned_addr learned my addr 172.21.15.5:0/116493083 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): get_auth_request con 0x7fc0180a36b0 auth_method 0 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 1 --2- 172.21.15.5:0/116493083 >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 0x7fc0180a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): get_auth_request con 0x7fc01806cc10 auth_method 0 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient(hunting): handle_auth_done global_id 74562 payload 274 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 1 -- 172.21.15.5:0/116493083 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381291213 0 0) 0x7fc01806d090 con 0x7fc0180a3fd0 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 -- 172.21.15.5:0/116493083 >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 msgr2=0x7fc0180a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 --2- 172.21.15.5:0/116493083 >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 0x7fc0180a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 -- 172.21.15.5:0/116493083 >> v1:172.21.15.5:6789/0 conn(0x7fc0180a3fd0 legacy=0x7fc018131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient: _finish_hunting 0 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 monclient: found mon.noname-a 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 -- 172.21.15.5:0/116493083 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180a43b0 con 0x7fc0180a36b0 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc00effd700 1 --2- 172.21.15.5:0/116493083 >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 0x7fc0180a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc0177fe700 1 --2- 172.21.15.5:0/116493083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0180a36b0 0x7fc01806f3b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc01806de50 tx=0x7fc00800acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a430d99077d16f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 1 -- 172.21.15.5:0/116493083 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00800cb70 con 0x7fc0180a36b0 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: dump: 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.926 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: _finish_auth 0 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 5 monclient: authenticate success, global_id 74562 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 1 -- 172.21.15.5:0/116493083 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00800ccd0 con 0x7fc0180a36b0 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:58.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient: get_monmap_and_config success 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 4 set_mon_vals no callback set 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 1 -- 172.21.15.5:0/116493083 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008018620 con 0x7fc0180a36b0 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc016ffd700 10 monclient: dump: 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.928 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 10 monclient: shutdown 2024-09-16T11:05:58.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- 172.21.15.5:0/116493083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0180a36b0 msgr2=0x7fc01806f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/116493083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0180a36b0 0x7fc01806f3b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc01806de50 tx=0x7fc00800acd0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- 172.21.15.5:0/116493083 shutdown_connections 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/116493083 >> v2:172.21.15.143:3300/0 conn(0x7fc01806cc10 0x7fc0180a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/116493083 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc0180a36b0 0x7fc01806f3b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- 172.21.15.5:0/116493083 >> 172.21.15.5:0/116493083 conn(0x7fc0180a12a0 msgr2=0x7fc0180a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- 172.21.15.5:0/116493083 shutdown_connections 2024-09-16T11:05:58.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.916+0000 7fc01cbc5700 1 -- 172.21.15.5:0/116493083 wait complete. 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: build_initial_monmap 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: monmap: 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:05:58.936 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 librados: starting msgr at 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 librados: starting objecter 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 Processor -- start 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- start start 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 librados: setting wanted keys 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 librados: calling monclient init 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: init 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _reopen_session rank -1 2024-09-16T11:05:58.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-a con 0x7fc01806cc10 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 0x7fc0181324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-c con 0x7fc0180a36b0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180a3fd0 0x7fc01811e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): picked mon.noname-b con 0x7fc0180a3fd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc018120db0 con 0x7fc01806cc10 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01809e130 con 0x7fc0180a3fd0 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): start opening mon connection 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180a30f0 con 0x7fc0180a36b0 2024-09-16T11:05:58.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): _renew_subs 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient(hunting): authenticate will time out at 1817.497870s 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 0x7fc0181324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52868/0 (socket says 172.21.15.5:52868) 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 1 -- 172.21.15.5:0/2677679725 learned_addr learned my addr 172.21.15.5:0/2677679725 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): get_auth_request con 0x7fc01806cc10 auth_method 0 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): get_auth_request con 0x7fc0180a36b0 auth_method 0 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180a3fd0 0x7fc01811e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): get_auth_request con 0x7fc0180a3fd0 auth_method 0 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:05:58.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc017fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient(hunting): handle_auth_done global_id 104108 payload 824 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 -- 172.21.15.5:0/2677679725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180a3fd0 msgr2=0x7fc01811e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180a3fd0 0x7fc01811e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 -- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 msgr2=0x7fc018131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient: _finish_hunting 0 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 monclient: found mon.noname-c 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 -- 172.21.15.5:0/2677679725 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0000094b0 con 0x7fc0180a36b0 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc0177fe700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:05:58.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc00effd700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 0x7fc0181324a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e980 tx=0x7fc00000ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 client.?.objecter ms_handle_connect 0x7fc0180a36b0 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000c690 con 0x7fc0180a36b0 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: dump: 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.942 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: _finish_auth 0 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00000c7f0 con 0x7fc0180a36b0 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 5 monclient: authenticate success, global_id 104108 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _renew_subs 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00800a650 con 0x7fc0180a36b0 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000cba0 con 0x7fc0180a36b0 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc014ff9700 10 monclient: dump: 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:05:58.944 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 client.104108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _renew_subs 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc015ffb700 4 set_mon_vals no callback set 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc018121e10 con 0x7fc0180a36b0 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.920+0000 7fc01cbc5700 1 librados: init done 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc01cbc5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc01cbc5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc01cbc5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:58.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc00000cd00 con 0x7fc0180a36b0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0180a3a90 con 0x7fc0180a36b0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 1 --2- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 0x7fbff8039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7fc000048440 con 0x7fc0180a36b0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 10 client.104108.objecter ms_dispatch 0x7fc018000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 3 client.104108.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 3 client.104108.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 20 client.104108.objecter dump_active .. 0 homeless 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc0177fe700 1 --2- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 0x7fbff8039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc0177fe700 10 monclient: get_auth_request con 0x7fbff8036f60 auth_method 0 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc014ff9700 10 client.104108.objecter ms_handle_connect 0x7fbff8036f60 2024-09-16T11:05:58.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.924+0000 7fc0177fe700 1 --2- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 0x7fbff8039420 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 crypto rx=0x7fc00800ad10 tx=0x7fc008012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:05:58.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.932+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc000011b00 con 0x7fc0180a36b0 2024-09-16T11:05:58.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.932+0000 7fc014ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:05:58.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:58.932+0000 7fc014ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:05:59.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc01cbc5700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T11:05:59.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc01cbc5700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T11:05:59.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc01cbc5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:05:59.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc0180008d0 con 0x7fc0180a36b0 2024-09-16T11:05:59.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=9,logm=60} crc {auth=1629025538,config=692619927,health=2877916515,kv=2723691402,logm=2364574708}) 2024-09-16T11:05:59.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc014ff9700 1 -- 172.21.15.5:0/2677679725 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc000017080 con 0x7fc0180a36b0 2024-09-16T11:05:59.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc014ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T11:05:59.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.172+0000 7fc014ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:05:59.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=3963591748,config=692619927,health=3306283535,kv=3001707193,logm=3929657231}) 2024-09-16T11:05:59.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=9,logm=60} crc {auth=1629025538,config=692619927,health=2877916515,kv=2723691402,logm=2364574708}) 2024-09-16T11:05:59.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.172+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=3270247777,config=692619927,health=3306283535,kv=3554031534,logm=239315841}) 2024-09-16T11:05:59.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528627365}) 2024-09-16T11:05:59.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912801187}) 2024-09-16T11:05:59.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528627365}) 2024-09-16T11:05:59.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.176+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358871586}) 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.176+0000 7fc01cbc5700 10 librados: watch_flush enter 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 10 librados: watch_flush exit 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 20 client.104108.objecter shutdown clearing up homeless session... 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 10 client.104108.objecter successfully canceled tick 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 msgr2=0x7fbff8039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 0x7fbff8039420 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 crypto rx=0x7fc00800ad10 tx=0x7fc008012000 comp rx=0 tx=0).stop 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 10 monclient: shutdown 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 msgr2=0x7fc0181324a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 0x7fc0181324a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc00000e980 tx=0x7fc00000ec90 comp rx=0 tx=0).stop 2024-09-16T11:05:59.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 shutdown_connections 2024-09-16T11:05:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180a3fd0 0x7fc01811e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc0180a36b0 0x7fc0181324a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:59.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> v2:172.21.15.143:6812/1098205034 conn(0x7fbff8036f60 0x7fbff8039420 unknown :-1 s=CLOSED pgs=3896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 --2- 172.21.15.5:0/2677679725 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc01806cc10 0x7fc018131f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:05:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 >> 172.21.15.5:0/2677679725 conn(0x7fc0180a12a0 msgr2=0x7fc01806d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:05:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 shutdown_connections 2024-09-16T11:05:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 -- 172.21.15.5:0/2677679725 wait complete. 2024-09-16T11:05:59.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:05:59.180+0000 7fc01cbc5700 1 librados: shutdown 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720571030}) 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267342365}) 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720571030}) 2024-09-16T11:05:59.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.180+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133675963}) 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338881489}) 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837936284}) 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338881489}) 2024-09-16T11:05:59.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.184+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226400735}) 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928919602}) 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170617189}) 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928919602}) 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258971844}) 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492931164}) 2024-09-16T11:05:59.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679526041}) 2024-09-16T11:05:59.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492931164}) 2024-09-16T11:05:59.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2725708211}) 2024-09-16T11:05:59.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079540601}) 2024-09-16T11:05:59.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988336119}) 2024-09-16T11:05:59.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079540601}) 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.188+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683619465}) 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290984466}) 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183028310}) 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290984466}) 2024-09-16T11:05:59.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729656243}) 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194091213}) 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=565258209}) 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194091213}) 2024-09-16T11:05:59.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679737990}) 2024-09-16T11:05:59.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469190321}) 2024-09-16T11:05:59.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495392437}) 2024-09-16T11:05:59.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469190321}) 2024-09-16T11:05:59.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.192+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123427561}) 2024-09-16T11:05:59.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T11:05:59.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246515254}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263489790}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246515254}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4141592822}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302846116}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3035455597}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302846116}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.196+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716996555}) 2024-09-16T11:05:59.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944993248}) 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1435427799}) 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944993248}) 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864479730}) 2024-09-16T11:05:59.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370817476}) 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238893274}) 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370817476}) 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1277832703}) 2024-09-16T11:05:59.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307252982}) 2024-09-16T11:05:59.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278047808}) 2024-09-16T11:05:59.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2307252982}) 2024-09-16T11:05:59.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.200+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247926873}) 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026112533}) 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2283033370}) 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026112533}) 2024-09-16T11:05:59.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1497413876}) 2024-09-16T11:05:59.208 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903461737}) 2024-09-16T11:05:59.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3287964325}) 2024-09-16T11:05:59.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903461737}) 2024-09-16T11:05:59.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909973158}) 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087865091}) 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185137117}) 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087865091}) 2024-09-16T11:05:59.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.204+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=696715774}) 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427089242}) 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793184678}) 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427089242}) 2024-09-16T11:05:59.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109884834}) 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178956398}) 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1785933181}) 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178956398}) 2024-09-16T11:05:59.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336486608}) 2024-09-16T11:05:59.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228894049}) 2024-09-16T11:05:59.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707067588}) 2024-09-16T11:05:59.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228894049}) 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.208+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207509921}) 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166512529}) 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841963978}) 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166512529}) 2024-09-16T11:05:59.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830238821}) 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815254116}) 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3406915934}) 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815254116}) 2024-09-16T11:05:59.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1321224611}) 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1889983316}) 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3962498764}) 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1889983316}) 2024-09-16T11:05:59.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477230396}) 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780961073}) 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526901975}) 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780961073}) 2024-09-16T11:05:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.212+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2030121500}) 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010759547}) 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421701151}) 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010759547}) 2024-09-16T11:05:59.219 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381777627}) 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29608898}) 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925985136}) 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=29608898}) 2024-09-16T11:05:59.220 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2951031294}) 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725856636}) 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340450268}) 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725856636}) 2024-09-16T11:05:59.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.216+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223925014}) 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105217380}) 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640516245}) 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105217380}) 2024-09-16T11:05:59.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089613158}) 2024-09-16T11:05:59.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789539233}) 2024-09-16T11:05:59.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122142518}) 2024-09-16T11:05:59.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789539233}) 2024-09-16T11:05:59.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792002495}) 2024-09-16T11:05:59.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837125549}) 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269384382}) 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837125549}) 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240164486}) 2024-09-16T11:05:59.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626995152}) 2024-09-16T11:05:59.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242957000}) 2024-09-16T11:05:59.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626995152}) 2024-09-16T11:05:59.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.220+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408687872}) 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781331450}) 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209458609}) 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781331450}) 2024-09-16T11:05:59.227 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550059393}) 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100876240}) 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493937139}) 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100876240}) 2024-09-16T11:05:59.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670757464}) 2024-09-16T11:05:59.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550907092}) 2024-09-16T11:05:59.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631521984}) 2024-09-16T11:05:59.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550907092}) 2024-09-16T11:05:59.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.224+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628862034}) 2024-09-16T11:05:59.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693956364}) 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=265363363}) 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693956364}) 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902805183}) 2024-09-16T11:05:59.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9768386}) 2024-09-16T11:05:59.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351782223}) 2024-09-16T11:05:59.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9768386}) 2024-09-16T11:05:59.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422624497}) 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462116973}) 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1203700306}) 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462116973}) 2024-09-16T11:05:59.233 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.228+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415122263}) 2024-09-16T11:05:59.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367926254}) 2024-09-16T11:05:59.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=999802399}) 2024-09-16T11:05:59.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367926254}) 2024-09-16T11:05:59.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260872320}) 2024-09-16T11:05:59.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808529194}) 2024-09-16T11:05:59.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59752706}) 2024-09-16T11:05:59.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1808529194}) 2024-09-16T11:05:59.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696632646}) 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223624548}) 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3508465291}) 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223624548}) 2024-09-16T11:05:59.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.232+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782472277}) 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399853058}) 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273229124}) 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399853058}) 2024-09-16T11:05:59.238 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2104844264}) 2024-09-16T11:05:59.239 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.239 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534416137}) 2024-09-16T11:05:59.239 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338605738}) 2024-09-16T11:05:59.240 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.240 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534416137}) 2024-09-16T11:05:59.240 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407756538}) 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160056873}) 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047137646}) 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160056873}) 2024-09-16T11:05:59.241 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.236+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3366869200}) 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176245236}) 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688799606}) 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176245236}) 2024-09-16T11:05:59.242 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648584315}) 2024-09-16T11:05:59.243 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.243 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983878952}) 2024-09-16T11:05:59.243 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2009140885}) 2024-09-16T11:05:59.244 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.244 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983878952}) 2024-09-16T11:05:59.244 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2736466072}) 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983718592}) 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2439923912}) 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983718592}) 2024-09-16T11:05:59.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.240+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304601039}) 2024-09-16T11:05:59.246 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.246 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121793374}) 2024-09-16T11:05:59.246 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199881016}) 2024-09-16T11:05:59.247 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.247 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121793374}) 2024-09-16T11:05:59.247 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3467278422}) 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254820388}) 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2724944460}) 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254820388}) 2024-09-16T11:05:59.248 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191416078}) 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270174339}) 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1298212179}) 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270174339}) 2024-09-16T11:05:59.249 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623599210}) 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289435800}) 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26414257}) 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289435800}) 2024-09-16T11:05:59.250 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.244+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1225347074}) 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047963482}) 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174622421}) 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047963482}) 2024-09-16T11:05:59.251 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970814094}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231339363}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321478781}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231339363}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311851696}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4224967458}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683370872}) 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.253 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4224967458}) 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.248+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2490619123}) 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546362577}) 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750043499}) 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.254 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546362577}) 2024-09-16T11:05:59.255 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033992626}) 2024-09-16T11:05:59.255 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.255 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965910460}) 2024-09-16T11:05:59.255 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896527221}) 2024-09-16T11:05:59.255 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.256 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965910460}) 2024-09-16T11:05:59.256 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075068491}) 2024-09-16T11:05:59.256 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.256 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114183474}) 2024-09-16T11:05:59.256 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955145530}) 2024-09-16T11:05:59.257 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.257 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114183474}) 2024-09-16T11:05:59.257 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2356318916}) 2024-09-16T11:05:59.257 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.257 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957207727}) 2024-09-16T11:05:59.258 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046620883}) 2024-09-16T11:05:59.258 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.258 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957207727}) 2024-09-16T11:05:59.258 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.252+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787414210}) 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699232008}) 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584973677}) 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699232008}) 2024-09-16T11:05:59.259 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783235503}) 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374633285}) 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4274284624}) 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374633285}) 2024-09-16T11:05:59.260 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673237630}) 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=1787133480,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419}) 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2156710225,mdsmap=1531151339,mgr=882151311}) 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=1787133480,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419}) 2024-09-16T11:05:59.262 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.256+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3161750443,mdsmap=2147943554,mgr=1566175811}) 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=10,logm=60} crc {auth=285229939,config=692619927,health=3264298073,kv=564032378,logm=1981435597}) 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=1995230390,config=692619927,health=3306283535,kv=2841592439,logm=826826166}) 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=10,kv=10,logm=60} crc {auth=285229939,config=692619927,health=3264298073,kv=564032378,logm=1981435597}) 2024-09-16T11:05:59.263 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=9,kv=10,logm=59} crc {auth=591643360,config=692619927,health=992128433,kv=564032378,logm=3618141146}) 2024-09-16T11:05:59.264 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.264 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411651261}) 2024-09-16T11:05:59.264 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139851172}) 2024-09-16T11:05:59.264 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.264 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411651261}) 2024-09-16T11:05:59.265 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756119530}) 2024-09-16T11:05:59.265 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.265 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401181855}) 2024-09-16T11:05:59.265 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505218320}) 2024-09-16T11:05:59.266 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.266 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401181855}) 2024-09-16T11:05:59.266 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.260+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950890822}) 2024-09-16T11:05:59.266 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.266 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058594451}) 2024-09-16T11:05:59.267 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878148012}) 2024-09-16T11:05:59.267 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.267 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3058594451}) 2024-09-16T11:05:59.267 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3853959380}) 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556524945}) 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=870617563}) 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556524945}) 2024-09-16T11:05:59.268 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579626015}) 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895783136}) 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410366997}) 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895783136}) 2024-09-16T11:05:59.269 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.264+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972927881}) 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496056436}) 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094511318}) 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496056436}) 2024-09-16T11:05:59.270 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815045620}) 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988856253}) 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741736829}) 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988856253}) 2024-09-16T11:05:59.271 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516419579}) 2024-09-16T11:05:59.272 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.272 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354478548}) 2024-09-16T11:05:59.272 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123859630}) 2024-09-16T11:05:59.272 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354478548}) 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635827914}) 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241943024}) 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043568555}) 2024-09-16T11:05:59.273 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.274 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241943024}) 2024-09-16T11:05:59.274 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.268+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321810825}) 2024-09-16T11:05:59.274 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.274 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630043299}) 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867752121}) 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630043299}) 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874584645}) 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.275 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3579213850}) 2024-09-16T11:05:59.276 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2914119215}) 2024-09-16T11:05:59.276 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.276 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3579213850}) 2024-09-16T11:05:59.276 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790721888}) 2024-09-16T11:05:59.276 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830999054}) 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417803464}) 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830999054}) 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1468564269}) 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.277 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484207108}) 2024-09-16T11:05:59.278 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374796336}) 2024-09-16T11:05:59.278 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.278 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484207108}) 2024-09-16T11:05:59.278 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.272+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560111403}) 2024-09-16T11:05:59.278 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721416459}) 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4028076775}) 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721416459}) 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492171745}) 2024-09-16T11:05:59.279 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023030546}) 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354109492}) 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023030546}) 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1001297384}) 2024-09-16T11:05:59.280 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.281 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829459068}) 2024-09-16T11:05:59.281 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3487230716}) 2024-09-16T11:05:59.281 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.281 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829459068}) 2024-09-16T11:05:59.281 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3677997463}) 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360369516}) 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353950362}) 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360369516}) 2024-09-16T11:05:59.282 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.276+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395012550}) 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373662473}) 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=275958312}) 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373662473}) 2024-09-16T11:05:59.283 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350316356}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501995694}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1469141541}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501995694}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842882453}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624002483}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1941211122}) 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.285 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624002483}) 2024-09-16T11:05:59.286 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.280+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931355041}) 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706126282}) 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956169131}) 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706126282}) 2024-09-16T11:05:59.287 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271153363}) 2024-09-16T11:05:59.288 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.289 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578721636}) 2024-09-16T11:05:59.289 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238497008}) 2024-09-16T11:05:59.289 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.289 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578721636}) 2024-09-16T11:05:59.289 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.284+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935268983}) 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002436835}) 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570304943}) 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002436835}) 2024-09-16T11:05:59.290 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120084680}) 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623067516}) 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224686277}) 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623067516}) 2024-09-16T11:05:59.292 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775274240}) 2024-09-16T11:05:59.293 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.293 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674307315}) 2024-09-16T11:05:59.294 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441999651}) 2024-09-16T11:05:59.294 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.294 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674307315}) 2024-09-16T11:05:59.294 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.288+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618073869}) 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788530534}) 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230757954}) 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788530534}) 2024-09-16T11:05:59.295 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451970625}) 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630553293}) 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917749538}) 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630553293}) 2024-09-16T11:05:59.297 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.292+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758706564}) 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523920411}) 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459596221}) 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523920411}) 2024-09-16T11:05:59.298 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749844911}) 2024-09-16T11:05:59.300 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.300 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934266119}) 2024-09-16T11:05:59.300 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803497358}) 2024-09-16T11:05:59.300 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.300 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934266119}) 2024-09-16T11:05:59.301 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.296+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973187230}) 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067159626}) 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=858986647}) 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067159626}) 2024-09-16T11:05:59.302 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3940475966}) 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633308878}) 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3742884299}) 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633308878}) 2024-09-16T11:05:59.304 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374230946}) 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3608548969}) 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963307708}) 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3608548969}) 2024-09-16T11:05:59.306 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.300+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340848000}) 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244706063}) 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390239268}) 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244706063}) 2024-09-16T11:05:59.307 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190799957}) 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996374161}) 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078011833}) 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996374161}) 2024-09-16T11:05:59.308 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3956075385}) 2024-09-16T11:05:59.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512019231}) 2024-09-16T11:05:59.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1740012761}) 2024-09-16T11:05:59.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.309 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512019231}) 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.304+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287464574}) 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255825243}) 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532242576}) 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.310 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255825243}) 2024-09-16T11:05:59.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029992782}) 2024-09-16T11:05:59.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674153160}) 2024-09-16T11:05:59.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369877564}) 2024-09-16T11:05:59.311 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674153160}) 2024-09-16T11:05:59.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332685921}) 2024-09-16T11:05:59.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081984491}) 2024-09-16T11:05:59.312 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251980172}) 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081984491}) 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456328844}) 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131779696}) 2024-09-16T11:05:59.313 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476815299}) 2024-09-16T11:05:59.314 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.314 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131779696}) 2024-09-16T11:05:59.314 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.308+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3672730118}) 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748960176}) 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333688683}) 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748960176}) 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195316291}) 2024-09-16T11:05:59.315 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532925786}) 2024-09-16T11:05:59.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2985299748}) 2024-09-16T11:05:59.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532925786}) 2024-09-16T11:05:59.316 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599229299}) 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3607370445}) 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031984938}) 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3607370445}) 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161872485}) 2024-09-16T11:05:59.317 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312857109}) 2024-09-16T11:05:59.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094473624}) 2024-09-16T11:05:59.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312857109}) 2024-09-16T11:05:59.318 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.312+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483670237}) 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071301652}) 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=173419360}) 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1071301652}) 2024-09-16T11:05:59.319 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227263129}) 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358489056}) 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1658160187}) 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358489056}) 2024-09-16T11:05:59.320 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743322789}) 2024-09-16T11:05:59.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651658610}) 2024-09-16T11:05:59.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705275336}) 2024-09-16T11:05:59.321 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651658610}) 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2580210345}) 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=732107520}) 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3224069584}) 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=732107520}) 2024-09-16T11:05:59.322 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.316+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108327970}) 2024-09-16T11:05:59.323 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.323 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337164656}) 2024-09-16T11:05:59.323 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002233285}) 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337164656}) 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535512438}) 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638787220}) 2024-09-16T11:05:59.324 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=834788248}) 2024-09-16T11:05:59.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638787220}) 2024-09-16T11:05:59.325 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=883974309}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631368611}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176477287}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631368611}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1934762390}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247216628}) 2024-09-16T11:05:59.326 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1904862603}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247216628}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.320+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232365531}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932757481}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014165286}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932757481}) 2024-09-16T11:05:59.327 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662977072}) 2024-09-16T11:05:59.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573579856}) 2024-09-16T11:05:59.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1200867125}) 2024-09-16T11:05:59.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.328 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573579856}) 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458263237}) 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660878876}) 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2394781814}) 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660878876}) 2024-09-16T11:05:59.329 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167706931}) 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028659949}) 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260136728}) 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028659949}) 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.324+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785348177}) 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.330 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1376318568}) 2024-09-16T11:05:59.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362802407}) 2024-09-16T11:05:59.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1376318568}) 2024-09-16T11:05:59.331 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609219724}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952747983}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593490215}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952747983}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774002796}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735313090}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=895597855}) 2024-09-16T11:05:59.332 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735313090}) 2024-09-16T11:05:59.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3736081525}) 2024-09-16T11:05:59.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255969204}) 2024-09-16T11:05:59.333 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756812003}) 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255969204}) 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.328+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464080163}) 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2777662912,mdsmap=1531151339,mgr=1566175811,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1176184986}) 2024-09-16T11:05:59.334 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=1749693955,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=2633483575}) 2024-09-16T11:05:59.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=84,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2777662912,mdsmap=1531151339,mgr=1566175811,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1176184986}) 2024-09-16T11:05:59.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=1609674578,mdsmap=1531151339,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=3212940092}) 2024-09-16T11:05:59.335 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=47,osdmap=28} crc {mgrstat=3874716972,mon_config_key=3443394649,monmap=1354390297,osd_metadata=3111584180,osd_pg_creating=759648051,osd_snap=3269701156,osdmap=4118880924}) 2024-09-16T11:05:59.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=6,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=48,osdmap=25} crc {mgrstat=1746481555,mon_config_key=4207273253,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=759648051,osd_snap=632565104,osdmap=266621027}) 2024-09-16T11:05:59.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=47,osdmap=28} crc {mgrstat=3874716972,mon_config_key=3443394649,monmap=1354390297,osd_metadata=3111584180,osd_pg_creating=759648051,osd_snap=3269701156,osdmap=4118880924}) 2024-09-16T11:05:59.336 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=48,osdmap=24} crc {mgrstat=3874716972,mon_config_key=3273021363,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=759648051,osd_snap=2861770897,osdmap=3880573970}) 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1834249380}) 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3436563023}) 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1834249380}) 2024-09-16T11:05:59.337 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=962316279}) 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=984635573}) 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2239505441}) 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=984635573}) 2024-09-16T11:05:59.338 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.332+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1633535754}) 2024-09-16T11:05:59.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.339 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1968044639}) 2024-09-16T11:05:59.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1076139327}) 2024-09-16T11:05:59.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1968044639}) 2024-09-16T11:05:59.340 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.336+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2414020001}) 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=1187983071}) 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=64} crc {osdmap=3397902286}) 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=1187983071}) 2024-09-16T11:05:59.345 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:05:59.340+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=67} crc {osdmap=1974038015}) 2024-09-16T11:06:00.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:06:00.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 10 monclient: get_monmap_and_config 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 10 monclient: monmap: 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:00.319701+0000 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:00.319701+0000 2024-09-16T11:06:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:00.322 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 1 Processor -- start 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 1 -- start start 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.316+0000 7fe26759e700 10 monclient: init 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe26809b3e0 0x7fe26809b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe26809b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:06:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe26809c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe26809bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2681326b0 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26809fd30 con 0x7fe2681326b0 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26809c620 con 0x7fe26809b3e0 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26809e350 con 0x7fe26809bd00 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient(hunting): authenticate will time out at 1818.896652s 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe26809b3e0 0x7fe26809b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe26809b3e0 0x7fe26809b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52870/0 (socket says 172.21.15.5:52870) 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 -- 172.21.15.5:0/722524969 learned_addr learned my addr 172.21.15.5:0/722524969 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:00.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): get_auth_request con 0x7fe26809b3e0 auth_method 0 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe26809c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): get_auth_request con 0x7fe26809bd00 auth_method 0 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient(hunting): handle_auth_done global_id 74571 payload 274 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 -- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 msgr2=0x7fe26809c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe26809c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 -- 172.21.15.5:0/722524969 >> v1:172.21.15.143:6789/0 conn(0x7fe2681326b0 legacy=0x7fe268132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 monclient: found mon.noname-a 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 -- 172.21.15.5:0/722524969 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26809ca20 con 0x7fe26809b3e0 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26659c700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26809b3e0 0x7fe26809b7c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe258009910 tx=0x7fe258009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c522ded1d41dbcc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 1 -- 172.21.15.5:0/722524969 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258011830 con 0x7fe26809b3e0 2024-09-16T11:06:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: dump: 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 1 -- 172.21.15.5:0/722524969 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258011990 con 0x7fe26809b3e0 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 5 monclient: authenticate success, global_id 74571 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 4 set_mon_vals no callback set 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 1 -- 172.21.15.5:0/722524969 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258011d40 con 0x7fe26809b3e0 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26559a700 10 monclient: dump: 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.331 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 10 monclient: shutdown 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26809b3e0 msgr2=0x7fe26809b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26809b3e0 0x7fe26809b7c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe258009910 tx=0x7fe258009c20 comp rx=0 tx=0).stop 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- 172.21.15.5:0/722524969 shutdown_connections 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe26809c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 --2- 172.21.15.5:0/722524969 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe26809b3e0 0x7fe26809b7c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- 172.21.15.5:0/722524969 >> 172.21.15.5:0/722524969 conn(0x7fe2680a45d0 msgr2=0x7fe2680a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- 172.21.15.5:0/722524969 shutdown_connections 2024-09-16T11:06:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.320+0000 7fe26759e700 1 -- 172.21.15.5:0/722524969 wait complete. 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient: monmap: 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:00.339 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 librados: starting msgr at 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 librados: starting objecter 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 Processor -- start 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 -- start start 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 librados: setting wanted keys 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 librados: calling monclient init 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient: init 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe26809b3e0 0x7fe2681cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe26809b3e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe2681cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe26809bd00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 0x7fe268121f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe2681326b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe268125bd0 con 0x7fe2681326b0 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680704a0 con 0x7fe26809b3e0 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2680a3450 con 0x7fe26809bd00 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26759e700 10 monclient(hunting): authenticate will time out at 1818.903274s 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe2681cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe2681cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37338/0 (socket says 172.21.15.5:37338) 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 0x7fe268121f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 1 -- 172.21.15.5:0/466379310 learned_addr learned my addr 172.21.15.5:0/466379310 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): get_auth_request con 0x7fe26809bd00 auth_method 0 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe26809b3e0 0x7fe2681cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): get_auth_request con 0x7fe2681326b0 auth_method 0 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): get_auth_request con 0x7fe26809b3e0 auth_method 0 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe266d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe265d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.324+0000 7fe26659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 10 monclient(hunting): handle_auth_done global_id 74574 payload 824 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 -- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 msgr2=0x7fe2681cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe2681cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 -- 172.21.15.5:0/466379310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe26809b3e0 msgr2=0x7fe2681cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe26809b3e0 0x7fe2681cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 monclient: found mon.noname-a 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 -- 172.21.15.5:0/466379310 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe258009570 con 0x7fe2681326b0 2024-09-16T11:06:00.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe266d9d700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 0x7fe268121f90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe26000cb90 tx=0x7fe26000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 client.?.objecter ms_handle_connect 0x7fe2681326b0 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260003c20 con 0x7fe2681326b0 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: dump: 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 5 monclient: authenticate success, global_id 74574 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe260003d80 con 0x7fe2681326b0 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _renew_subs 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26809b0d0 con 0x7fe2681326b0 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26000aa80 con 0x7fe2681326b0 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 client.74574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe264d99700 4 set_mon_vals no callback set 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe256ffd700 10 monclient: dump: 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.347 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _renew_subs 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe268125eb0 con 0x7fe2681326b0 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 1 librados: init done 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.328+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26809b7c0 con 0x7fe2681326b0 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe26000abe0 con 0x7fe2681326b0 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 1 --2- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 0x7fe248039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(660..660 src has 395..660) v4 ==== 5783+0+0 (secure 0 0 0) 0x7fe260048a80 con 0x7fe2681326b0 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 10 client.74574.objecter ms_dispatch 0x7fe268000fd0 osd_map(660..660 src has 395..660) v4 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 3 client.74574.objecter handle_osd_map got epochs [660,660] > 0 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 3 client.74574.objecter handle_osd_map decoding full epoch 660 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 20 client.74574.objecter dump_active .. 0 homeless 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe26659c700 1 --2- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 0x7fe248039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe26659c700 10 monclient: get_auth_request con 0x7fe248036f90 auth_method 0 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe256ffd700 10 client.74574.objecter ms_handle_connect 0x7fe248036f90 2024-09-16T11:06:00.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.332+0000 7fe26659c700 1 --2- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 0x7fe248039450 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 crypto rx=0x7fe258000c00 tx=0x7fe258008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.336+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe260011950 con 0x7fe2681326b0 2024-09-16T11:06:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.336+0000 7fe256ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.336+0000 7fe256ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:00.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:06:00.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:06:00.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe26809e010 con 0x7fe2681326b0 2024-09-16T11:06:00.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe256ffd700 1 -- 172.21.15.5:0/466379310 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fe260011380 con 0x7fe2681326b0 2024-09-16T11:06:00.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe256ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:06:00.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe256ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:00.581 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:06:00.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 librados: watch_flush enter 2024-09-16T11:06:00.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 librados: watch_flush exit 2024-09-16T11:06:00.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 20 client.74574.objecter shutdown clearing up homeless session... 2024-09-16T11:06:00.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 client.74574.objecter successfully canceled tick 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 msgr2=0x7fe248039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 0x7fe248039450 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 crypto rx=0x7fe258000c00 tx=0x7fe258008030 comp rx=0 tx=0).stop 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 10 monclient: shutdown 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 msgr2=0x7fe268121f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.576+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 0x7fe268121f90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe26000cb90 tx=0x7fe26000cea0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 shutdown_connections 2024-09-16T11:06:00.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe248036f90 0x7fe248039450 unknown :-1 s=CLOSED pgs=3897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe2681326b0 0x7fe268121f90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe26809bd00 0x7fe2681cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 --2- 172.21.15.5:0/466379310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe26809b3e0 0x7fe2681cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 >> 172.21.15.5:0/466379310 conn(0x7fe2680a45d0 msgr2=0x7fe268131f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 shutdown_connections 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 -- 172.21.15.5:0/466379310 wait complete. 2024-09-16T11:06:00.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.580+0000 7fe26759e700 1 librados: shutdown 2024-09-16T11:06:00.594 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:06:00.594 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T11:06:00.594 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:06:00.594 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: get_monmap_and_config 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: monmap: 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:00.713045+0000 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:00.713045+0000 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.718 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 Processor -- start 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 -- start start 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: init 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f435c135c30 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 0x7f435c13aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-c con 0x7f435c0a04f0 addr v2:172.21.15.143:3300/0 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-b con 0x7f435c13afb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f435c135750 con 0x7f435c135c30 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435c09f830 con 0x7f435c0a04f0 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c09f1c0 con 0x7f435c13afb0 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f4360a39700 10 monclient(hunting): authenticate will time out at 1819.290170s 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 0x7f435c13aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 0x7f435c13aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.5:41192/0 (socket says 172.21.15.5:41192) 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37340/0 (socket says 172.21.15.5:37340) 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435b7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f435c135c30 0x7f435c136010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.5:51460/0 (socket says 172.21.15.5:51460) 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 1 -- 172.21.15.5:0/1525516061 learned_addr learned my addr 172.21.15.5:0/1525516061 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): get_auth_request con 0x7f435c0a04f0 auth_method 0 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): get_auth_request con 0x7f435c13afb0 auth_method 0 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient(hunting): handle_auth_done global_id 104117 payload 274 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 -- 172.21.15.5:0/1525516061 >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 msgr2=0x7f435c13aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 --2- 172.21.15.5:0/1525516061 >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 0x7f435c13aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 -- 172.21.15.5:0/1525516061 >> v1:172.21.15.143:6789/0 conn(0x7f435c135c30 legacy=0x7f435c136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 monclient: found mon.noname-b 2024-09-16T11:06:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:06:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 -- 172.21.15.5:0/1525516061 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c071b30 con 0x7f435c13afb0 2024-09-16T11:06:00.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.712+0000 7f435bfff700 1 --2- 172.21.15.5:0/1525516061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4354009d90 tx=0x7f435400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd32d19431f261b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 1 -- 172.21.15.5:0/1525516061 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435400e070 con 0x7f435c13afb0 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: dump: 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.730 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 1 -- 172.21.15.5:0/1525516061 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435400bb20 con 0x7f435c13afb0 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f4360a39700 5 monclient: authenticate success, global_id 104117 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f4360a39700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:00.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f4360a39700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 1 -- 172.21.15.5:0/1525516061 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354012430 con 0x7f435c13afb0 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f4360a39700 4 set_mon_vals no callback set 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.716+0000 7f435a7fc700 10 monclient: dump: 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.732 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: shutdown 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- 172.21.15.5:0/1525516061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 msgr2=0x7f435c13d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- 172.21.15.5:0/1525516061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4354009d90 tx=0x7f435400b330 comp rx=0 tx=0).stop 2024-09-16T11:06:00.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- 172.21.15.5:0/1525516061 shutdown_connections 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- 172.21.15.5:0/1525516061 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c13afb0 0x7f435c13d3a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- 172.21.15.5:0/1525516061 >> v2:172.21.15.143:3300/0 conn(0x7f435c0a04f0 0x7f435c13aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- 172.21.15.5:0/1525516061 >> 172.21.15.5:0/1525516061 conn(0x7f435c0a45d0 msgr2=0x7f435c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- 172.21.15.5:0/1525516061 shutdown_connections 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- 172.21.15.5:0/1525516061 wait complete. 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: monmap: 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.734 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 librados: starting msgr at 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 librados: starting objecter 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 Processor -- start 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- start start 2024-09-16T11:06:00.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 librados: setting wanted keys 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 librados: calling monclient init 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: init 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 0x7f435c134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-c con 0x7f435c0a04f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c135c30 0x7f435c134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-a con 0x7f435c135c30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f435c13afb0 0x7f435c11b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): picked mon.noname-b con 0x7f435c13afb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c0a33a0 con 0x7f435c135c30 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c13a940 con 0x7f435c13afb0 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435c11f060 con 0x7f435c0a04f0 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4360a39700 10 monclient(hunting): authenticate will time out at 1819.298862s 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 0x7f435c134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 0x7f435c134110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37346/0 (socket says 172.21.15.5:37346) 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 -- 172.21.15.5:0/3165271542 learned_addr learned my addr 172.21.15.5:0/3165271542 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): get_auth_request con 0x7f435c0a04f0 auth_method 0 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c135c30 0x7f435c134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): get_auth_request con 0x7f435c135c30 auth_method 0 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f435c13afb0 0x7f435c11b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): get_auth_request con 0x7f435c13afb0 auth_method 0 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient(hunting): handle_auth_done global_id 104120 payload 824 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f435c13afb0 msgr2=0x7f435c11b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f435c13afb0 0x7f435c11b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c135c30 msgr2=0x7f435c134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c135c30 0x7f435c134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 monclient: found mon.noname-c 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354009a70 con 0x7f435c0a04f0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c135c30 0x7f435c134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f435b7fe700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 0x7f435c134110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f434c00c700 tx=0x7f434c00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4343fff700 10 client.?.objecter ms_handle_connect 0x7f435c0a04f0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.720+0000 7f4343fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c005cd0 con 0x7f435c0a04f0 2024-09-16T11:06:00.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: dump: 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.742 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 5 monclient: authenticate success, global_id 104120 2024-09-16T11:06:00.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434c005e30 con 0x7f435c0a04f0 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434c0072d0 con 0x7f435c0a04f0 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 monclient: _renew_subs 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435c11d650 con 0x7f435c0a04f0 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4359ffb700 4 set_mon_vals no callback set 2024-09-16T11:06:00.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: dump: 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 client.104120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:00.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 monclient: _renew_subs 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435c11daf0 con 0x7f435c0a04f0 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 1 librados: init done 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 0x7f435c11b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c067380 con 0x7f435c11b5a0 2024-09-16T11:06:00.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f434c007430 con 0x7f435c0a04f0 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 0x7f435c11b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient: get_auth_request con 0x7f435c11b5a0 auth_method 0 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 --2- 172.21.15.5:0/3165271542 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4348036f60 0x7f4348039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(661..661 src has 395..661) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f434c0472b0 con 0x7f435c0a04f0 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 client.104120.objecter ms_dispatch 0x7f435c000fd0 osd_map(661..661 src has 395..661) v4 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 3 client.104120.objecter handle_osd_map got epochs [661,661] > 0 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 3 client.104120.objecter handle_osd_map decoding full epoch 661 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 10 monclient(hunting): handle_auth_done global_id 74580 payload 824 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435c071880 con 0x7f435c11b5a0 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 20 client.104120.objecter dump_active .. 0 homeless 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 client.104120.objecter ms_handle_connect 0x7f435c11b5a0 2024-09-16T11:06:00.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 0x7f435c11b980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f435000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 client.104120.objecter resend_mon_ops 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435bfff700 1 --2- 172.21.15.5:0/3165271542 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4348036f60 0x7f4348039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435bfff700 10 monclient: get_auth_request con 0x7f4348036f60 auth_method 0 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435000fcc0 con 0x7f435c11b5a0 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: dump: 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.749 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f435000fe20 con 0x7f435c11b5a0 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 msgr2=0x7f435c11b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 0x7f435c11b980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f435000f520 comp rx=0 tx=0).stop 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 msgr2=0x7f435c11b980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11b5a0 0x7f435c11b980 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f435000f520 comp rx=0 tx=0).stop 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f4343fff700 10 client.104120.objecter ms_handle_connect 0x7f4348036f60 2024-09-16T11:06:00.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.724+0000 7f435bfff700 1 --2- 172.21.15.5:0/3165271542 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4348036f60 0x7f4348039420 secure :-1 s=READY pgs=3898 cs=0 l=1 rev1=1 crypto rx=0x7f4354014040 tx=0x7f4354011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.732+0000 7f4360a39700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:06:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.732+0000 7f4360a39700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 0x7f435c137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.732+0000 7f4360a39700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.732+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435c11f1d0 con 0x7f435c11bec0 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 0x7f435c137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient: get_auth_request con 0x7f435c11bec0 auth_method 0 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 10 monclient(hunting): handle_auth_done global_id 74583 payload 824 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 1 -- 172.21.15.5:0/3165271542 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f435c136010 con 0x7f435c11bec0 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 client.104120.objecter ms_handle_connect 0x7f435c11bec0 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435affd700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 0x7f435c137810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f4350004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 client.104120.objecter resend_mon_ops 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350014850 con 0x7f435c11bec0 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 monclient: dump: 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.754 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4779 (secure 0 0 0) 0x7f435001c3b0 con 0x7f435c11bec0 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 msgr2=0x7f435c137810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 0x7f435c137810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f4350004770 comp rx=0 tx=0).stop 2024-09-16T11:06:00.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 msgr2=0x7f435c137810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4343fff700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f435c11bec0 0x7f435c137810 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4350006d70 tx=0x7f4350004770 comp rx=0 tx=0).stop 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:06:00.756 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 7, 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.757 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.758 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:00.759 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:06:00.760 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:06:00.760 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:06:00.760 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.761 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.762 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.762 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.762 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.762 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:06:00.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.763 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:06:00.764 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:06:00.765 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.766 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.767 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.768 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:06:00.769 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:00.770 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.771 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.771 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.771 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:06:00.771 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 10 librados: watch_flush enter 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 10 librados: watch_flush exit 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 20 client.104120.objecter shutdown clearing up homeless session... 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 10 client.104120.objecter successfully canceled tick 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4348036f60 msgr2=0x7f4348039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 --2- 172.21.15.5:0/3165271542 >> v2:172.21.15.143:6812/1098205034 conn(0x7f4348036f60 0x7f4348039420 secure :-1 s=READY pgs=3898 cs=0 l=1 rev1=1 crypto rx=0x7f4354014040 tx=0x7f4354011000 comp rx=0 tx=0).stop 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 10 monclient: shutdown 2024-09-16T11:06:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 msgr2=0x7f435c134110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 --2- 172.21.15.5:0/3165271542 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f435c0a04f0 0x7f435c134110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f434c00c700 tx=0x7f434c00ca10 comp rx=0 tx=0).stop 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435bfff700 1 -- 172.21.15.5:0/3165271542 reap_dead start 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f435bfff700 1 -- 172.21.15.5:0/3165271542 reap_dead start 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 shutdown_connections 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 >> 172.21.15.5:0/3165271542 conn(0x7f435c0a45d0 msgr2=0x7f435c09c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 shutdown_connections 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 -- 172.21.15.5:0/3165271542 wait complete. 2024-09-16T11:06:00.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.736+0000 7f4360a39700 1 librados: shutdown 2024-09-16T11:06:00.774 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:06:00.775 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:06:00.775 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:06:00.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 10 monclient: get_monmap_and_config 2024-09-16T11:06:00.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 10 monclient: monmap: 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:00.895289+0000 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:00.895289+0000 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:00.897 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:00.898 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 1 Processor -- start 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 1 -- start start 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.892+0000 7f82a3cf2700 10 monclient: init 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T11:06:00.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f829c0a2eb0 0x7f829c0a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-c con 0x7f829c0a2eb0 addr v2:172.21.15.143:3300/0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f829c0a4280 addr v1:172.21.15.5:6789/0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f829c13aeb0 0x7f829c13d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-a con 0x7f829c13aeb0 addr v2:172.21.15.5:3300/0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c09fdc0 con 0x7f829c0a4280 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c0a1e50 con 0x7f829c13aeb0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f829c09fae0 con 0x7f829c0a2eb0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient(hunting): authenticate will time out at 1819.472296s 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f829c13aeb0 0x7f829c13d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a128d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f829c0a4280 0x7f829c0a37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33050/0 (socket says 172.21.15.5:33050) 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): get_auth_request con 0x7f829c13aeb0 auth_method 0 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a128d700 1 -- 172.21.15.5:0/2289428734 learned_addr learned my addr 172.21.15.5:0/2289428734 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/2289428734 >> v2:172.21.15.143:3300/0 conn(0x7f829c0a2eb0 0x7f829c0a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request con 0x7f829c0a2eb0 auth_method 0 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a1a8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a1a8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient(hunting): handle_auth_done global_id 74586 payload 274 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 -- 172.21.15.5:0/2289428734 >> v2:172.21.15.143:3300/0 conn(0x7f829c0a2eb0 msgr2=0x7f829c0a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 --2- 172.21.15.5:0/2289428734 >> v2:172.21.15.143:3300/0 conn(0x7f829c0a2eb0 0x7f829c0a3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 -- 172.21.15.5:0/2289428734 >> v1:172.21.15.5:6789/0 conn(0x7f829c0a4280 legacy=0x7f829c0a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 monclient: found mon.noname-a 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 -- 172.21.15.5:0/2289428734 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c09efb0 con 0x7f829c13aeb0 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a228f700 1 --2- 172.21.15.5:0/2289428734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c13aeb0 0x7f829c13d2a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f829c0a6070 tx=0x7f829400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d2422ba6a41d94 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 1 -- 172.21.15.5:0/2289428734 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400e070 con 0x7f829c13aeb0 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: dump: 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 5 monclient: authenticate success, global_id 74586 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 1 -- 172.21.15.5:0/2289428734 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829400bbd0 con 0x7f829c13aeb0 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 4 set_mon_vals no callback set 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 1 -- 172.21.15.5:0/2289428734 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294012430 con 0x7f829c13aeb0 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a0a8c700 10 monclient: dump: 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.907 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 10 monclient: shutdown 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- 172.21.15.5:0/2289428734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c13aeb0 msgr2=0x7f829c13d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/2289428734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c13aeb0 0x7f829c13d2a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f829c0a6070 tx=0x7f829400b330 comp rx=0 tx=0).stop 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- 172.21.15.5:0/2289428734 shutdown_connections 2024-09-16T11:06:00.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/2289428734 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c13aeb0 0x7f829c13d2a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/2289428734 >> v2:172.21.15.143:3300/0 conn(0x7f829c0a2eb0 0x7f829c0a3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- 172.21.15.5:0/2289428734 >> 172.21.15.5:0/2289428734 conn(0x7f829c0a4fb0 msgr2=0x7f829c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- 172.21.15.5:0/2289428734 shutdown_connections 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.896+0000 7f82a3cf2700 1 -- 172.21.15.5:0/2289428734 wait complete. 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: build_initial_monmap 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: monmap: 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:00.909 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:00.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:00.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:00.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 librados: starting msgr at 2024-09-16T11:06:00.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 librados: starting objecter 2024-09-16T11:06:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 Processor -- start 2024-09-16T11:06:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- start start 2024-09-16T11:06:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 librados: setting wanted keys 2024-09-16T11:06:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 librados: calling monclient init 2024-09-16T11:06:00.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: init 2024-09-16T11:06:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:06:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f829c0a2eb0 0x7f829c139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-b con 0x7f829c0a2eb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 0x7f829c139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-a con 0x7f829c0a4280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c13aeb0 0x7f829c1208b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): picked mon.noname-c con 0x7f829c13aeb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c1242b0 con 0x7f829c0a4280 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c09f6a0 con 0x7f829c0a2eb0 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0a7340 con 0x7f829c13aeb0 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:00.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient(hunting): authenticate will time out at 1819.477812s 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 0x7f829c139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 0x7f829c139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52924/0 (socket says 172.21.15.5:52924) 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 -- 172.21.15.5:0/1423792588 learned_addr learned my addr 172.21.15.5:0/1423792588 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): get_auth_request con 0x7f829c0a4280 auth_method 0 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c13aeb0 0x7f829c1208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f829c0a2eb0 0x7f829c139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): get_auth_request con 0x7f829c13aeb0 auth_method 0 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request con 0x7f829c0a2eb0 auth_method 0 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a228f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient(hunting): handle_auth_done global_id 74592 payload 824 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c13aeb0 msgr2=0x7f829c1208b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c13aeb0 0x7f829c1208b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f829c0a2eb0 msgr2=0x7f829c139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f829c0a2eb0 0x7f829c139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient: _finish_hunting 0 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 monclient: found mon.noname-a 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8294009a70 con 0x7f829c0a4280 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f829c0a2eb0 0x7f829c139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a128d700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 0x7f829c139eb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f829800c730 tx=0x7f829800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 client.?.objecter ms_handle_connect 0x7f829c0a4280 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298005d90 con 0x7f829c0a4280 2024-09-16T11:06:00.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: dump: 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.924 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: _finish_auth 0 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: _check_auth_tickets 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 5 monclient: authenticate success, global_id 74592 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82980070d0 con 0x7f829c0a4280 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298007440 con 0x7f829c0a4280 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:00.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f8293fff700 4 set_mon_vals no callback set 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82927fc700 10 monclient: dump: 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _renew_subs 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c0a6ff0 con 0x7f829c0a4280 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 client.74592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _renew_subs 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c122d80 con 0x7f829c0a4280 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.900+0000 7f82a3cf2700 1 librados: init done 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f82980075a0 con 0x7f829c0a4280 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a3cf2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 0x7f829c02b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c067b80 con 0x7f829c1215c0 2024-09-16T11:06:00.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 --2- 172.21.15.5:0/1423792588 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8280036f60 0x7f8280039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 0x7f829c02b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient: get_auth_request con 0x7f829c1215c0 auth_method 0 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(661..661 src has 395..661) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f82980474a0 con 0x7f829c0a4280 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 client.74592.objecter ms_dispatch 0x7f829c000fd0 osd_map(661..661 src has 395..661) v4 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 3 client.74592.objecter handle_osd_map got epochs [661,661] > 0 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 3 client.74592.objecter handle_osd_map decoding full epoch 661 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 20 client.74592.objecter dump_active .. 0 homeless 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_done global_id 104126 payload 824 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f829c0a5d80 con 0x7f829c1215c0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a228f700 1 --2- 172.21.15.5:0/1423792588 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8280036f60 0x7f8280039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a228f700 10 monclient: get_auth_request con 0x7f8280036f60 auth_method 0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 client.74592.objecter ms_handle_connect 0x7f829c1215c0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 client.74592.objecter resend_mon_ops 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 0x7f829c02b6a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f828c010830 tx=0x7f828c010bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c004020 con 0x7f829c1215c0 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 monclient: dump: 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.929 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 client.74592.objecter ms_handle_connect 0x7f8280036f60 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82a228f700 1 --2- 172.21.15.5:0/1423792588 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8280036f60 0x7f8280039420 secure :-1 s=READY pgs=3899 cs=0 l=1 rev1=1 crypto rx=0x7f8294014040 tx=0x7f8294009f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f828c00c050 con 0x7f829c1215c0 2024-09-16T11:06:00.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 msgr2=0x7f829c02b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 0x7f829c02b6a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f828c010830 tx=0x7f828c010bf0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 msgr2=0x7f829c02b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.904+0000 7f82927fc700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1215c0 0x7f829c02b6a0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f828c010830 tx=0x7f828c010bf0 comp rx=0 tx=0).stop 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 0x7f829c02bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c1240f0 con 0x7f829c1219a0 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 0x7f829c02bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient: get_auth_request con 0x7f829c1219a0 auth_method 0 2024-09-16T11:06:00.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 10 monclient(hunting): handle_auth_done global_id 104129 payload 824 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 1 -- 172.21.15.5:0/1423792588 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f829c1247f0 con 0x7f829c1219a0 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 client.74592.objecter ms_handle_connect 0x7f829c1219a0 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 client.74592.objecter resend_mon_ops 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a1a8e700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 0x7f829c02bbe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f828c01a000 tx=0x7f828c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c020220 con 0x7f829c1219a0 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 monclient: dump: 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f828c003740 con 0x7f829c1219a0 2024-09-16T11:06:00.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 msgr2=0x7f829c02bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 0x7f829c02bbe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f828c01a000 tx=0x7f828c004770 comp rx=0 tx=0).stop 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 msgr2=0x7f829c02bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82927fc700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f829c1219a0 0x7f829c02bbe0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f828c01a000 tx=0x7f828c004770 comp rx=0 tx=0).stop 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:00.934 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 7, 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.935 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.936 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:06:00.937 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:00.938 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:06:00.939 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.940 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.941 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.942 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:00.943 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.944 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:06:00.946 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:06:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 librados: watch_flush enter 2024-09-16T11:06:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 librados: watch_flush exit 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 20 client.74592.objecter shutdown clearing up homeless session... 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 client.74592.objecter successfully canceled tick 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8280036f60 msgr2=0x7f8280039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/1423792588 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8280036f60 0x7f8280039420 secure :-1 s=READY pgs=3899 cs=0 l=1 rev1=1 crypto rx=0x7f8294014040 tx=0x7f8294009f90 comp rx=0 tx=0).stop 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 10 monclient: shutdown 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 msgr2=0x7f829c139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a3cf2700 1 --2- 172.21.15.5:0/1423792588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f829c0a4280 0x7f829c139eb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f829800c730 tx=0x7f829800ca40 comp rx=0 tx=0).stop 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.912+0000 7f82a228f700 1 -- 172.21.15.5:0/1423792588 reap_dead start 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a228f700 1 -- 172.21.15.5:0/1423792588 reap_dead start 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 shutdown_connections 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 >> 172.21.15.5:0/1423792588 conn(0x7f829c0a4fb0 msgr2=0x7f829c13bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:00.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 shutdown_connections 2024-09-16T11:06:00.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a3cf2700 1 -- 172.21.15.5:0/1423792588 wait complete. 2024-09-16T11:06:00.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:00.916+0000 7f82a3cf2700 1 librados: shutdown 2024-09-16T11:06:00.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:06:00.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:06:00.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:06:01.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: get_monmap_and_config 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: build_initial_monmap 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: monmap: 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:01.070329+0000 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:01.070329+0000 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:01.072 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:01.073 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:01.073 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 Processor -- start 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 -- start start 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: init 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2adc0a2eb0 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-b con 0x7f2adc0a4280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:01.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2adc13aeb0 0x7f2adc13d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-c con 0x7f2adc13aeb0 addr v2:172.21.15.143:3300/0 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc09fdc0 con 0x7f2adc0a2eb0 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0a1e50 con 0x7f2adc13aeb0 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc09fae0 con 0x7f2adc0a4280 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ae0bdb700 10 monclient(hunting): authenticate will time out at 1819.647679s 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.068+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37376/0 (socket says 172.21.15.5:37376) 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 -- 172.21.15.5:0/2255475269 learned_addr learned my addr 172.21.15.5:0/2255475269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 1 --2- 172.21.15.5:0/2255475269 >> v2:172.21.15.143:3300/0 conn(0x7f2adc13aeb0 0x7f2adc13d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): get_auth_request con 0x7f2adc0a4280 auth_method 0 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): get_auth_request con 0x7f2adc13aeb0 auth_method 0 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_done global_id 104132 payload 274 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 -- 172.21.15.5:0/2255475269 >> v2:172.21.15.143:3300/0 conn(0x7f2adc13aeb0 msgr2=0x7f2adc13d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 --2- 172.21.15.5:0/2255475269 >> v2:172.21.15.143:3300/0 conn(0x7f2adc13aeb0 0x7f2adc13d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 -- 172.21.15.5:0/2255475269 >> v1:172.21.15.143:6789/0 conn(0x7f2adc0a2eb0 legacy=0x7f2adc0a3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient: _finish_hunting 0 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 monclient: found mon.noname-b 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 -- 172.21.15.5:0/2255475269 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc09efb0 con 0x7f2adc0a4280 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad3fff700 1 --2- 172.21.15.5:0/2255475269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009910 tx=0x7f2ac4009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba45824969fdb7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 1 -- 172.21.15.5:0/2255475269 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4011830 con 0x7f2adc0a4280 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: dump: 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: _finish_auth 0 2024-09-16T11:06:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: _check_auth_tickets 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 1 -- 172.21.15.5:0/2255475269 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4011990 con 0x7f2adc0a4280 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 5 monclient: authenticate success, global_id 104132 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 1 -- 172.21.15.5:0/2255475269 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4011d40 con 0x7f2adc0a4280 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 4 set_mon_vals no callback set 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ad9d9b700 10 monclient: dump: 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.082 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 10 monclient: shutdown 2024-09-16T11:06:01.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:01.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/2255475269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 msgr2=0x7f2adc0a39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/2255475269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009910 tx=0x7f2ac4009c20 comp rx=0 tx=0).stop 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/2255475269 shutdown_connections 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/2255475269 >> v2:172.21.15.143:3300/0 conn(0x7f2adc13aeb0 0x7f2adc13d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/2255475269 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc0a4280 0x7f2adc0a39a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/2255475269 >> 172.21.15.5:0/2255475269 conn(0x7f2adc0a4fb0 msgr2=0x7f2adc0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/2255475269 shutdown_connections 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.072+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/2255475269 wait complete. 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: build_initial_monmap 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: monmap: 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 librados: starting msgr at 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 librados: starting objecter 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 Processor -- start 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- start start 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 librados: setting wanted keys 2024-09-16T11:06:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 librados: calling monclient init 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: init 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc0a2eb0 0x7f2adc139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-b con 0x7f2adc0a2eb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 0x7f2adc13a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-a con 0x7f2adc0a4280 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): picked mon.noname-c con 0x7f2adc13aeb0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0a7830 con 0x7f2adc0a4280 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0a7390 con 0x7f2adc0a2eb0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc09f6a0 con 0x7f2adc13aeb0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient(hunting): authenticate will time out at 1819.654083s 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 0x7f2adc13a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:37384/0 (socket says 172.21.15.5:37384) 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 0x7f2adc13a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52936/0 (socket says 172.21.15.5:52936) 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc0a2eb0 0x7f2adc139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 1 -- 172.21.15.5:0/51573313 learned_addr learned my addr 172.21.15.5:0/51573313 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): get_auth_request con 0x7f2adc0a4280 auth_method 0 2024-09-16T11:06:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): get_auth_request con 0x7f2adc13aeb0 auth_method 0 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): get_auth_request con 0x7f2adc0a2eb0 auth_method 0 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient(hunting): handle_auth_done global_id 104135 payload 824 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc0a2eb0 msgr2=0x7f2adc139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc0a2eb0 0x7f2adc139cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 msgr2=0x7f2adc13a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 0x7f2adc13a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient: _finish_hunting 0 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 monclient: found mon.noname-c 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4009570 con 0x7f2adc13aeb0 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad3fff700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc0a4280 0x7f2adc13a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2adad9d700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400efc0 tx=0x7f2ad4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:01.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 client.?.objecter ms_handle_connect 0x7f2adc13aeb0 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400ca60 con 0x7f2adc13aeb0 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: dump: 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.096 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: _finish_auth 0 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 5 monclient: authenticate success, global_id 104135 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad400f040 con 0x7f2adc13aeb0 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad400ce20 con 0x7f2adc13aeb0 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _renew_subs 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc0a4b00 con 0x7f2adc13aeb0 2024-09-16T11:06:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad959a700 4 set_mon_vals no callback set 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ad2ffd700 10 monclient: dump: 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.098 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 client.104135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _renew_subs 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc122d90 con 0x7f2adc13aeb0 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.076+0000 7f2ae0bdb700 1 librados: init done 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ae0bdb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 0x7f2adc121110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc067b80 con 0x7f2adc120d30 2024-09-16T11:06:01.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f2ad4014480 con 0x7f2adc13aeb0 2024-09-16T11:06:01.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 0x7f2adc121110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 --2- 172.21.15.5:0/51573313 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2abc036f60 0x7f2abc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient: get_auth_request con 0x7f2adc120d30 auth_method 0 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(661..661 src has 395..661) v4 ==== 5783+0+0 (secure 0 0 0) 0x7f2ad40489c0 con 0x7f2adc13aeb0 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 client.104135.objecter ms_dispatch 0x7f2adc000fd0 osd_map(661..661 src has 395..661) v4 2024-09-16T11:06:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 3 client.104135.objecter handle_osd_map got epochs [661,661] > 0 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 3 client.104135.objecter handle_osd_map decoding full epoch 661 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 20 client.104135.objecter dump_active .. 0 homeless 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad3fff700 1 --2- 172.21.15.5:0/51573313 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2abc036f60 0x7f2abc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad3fff700 10 monclient: get_auth_request con 0x7f2abc036f60 auth_method 0 2024-09-16T11:06:01.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 10 monclient(hunting): handle_auth_done global_id 94285 payload 824 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2adc123f90 con 0x7f2adc120d30 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ada59c700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 0x7f2adc121110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00f770 tx=0x7f2acc00fb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 client.104135.objecter ms_handle_connect 0x7f2adc120d30 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 client.104135.objecter resend_mon_ops 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0052e0 con 0x7f2adc120d30 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 monclient: dump: 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 client.104135.objecter ms_handle_connect 0x7f2abc036f60 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad3fff700 1 --2- 172.21.15.5:0/51573313 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2abc036f60 0x7f2abc039420 secure :-1 s=READY pgs=3900 cs=0 l=1 rev1=1 crypto rx=0x7f2adc122040 tx=0x7f2ac4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2acc005440 con 0x7f2adc120d30 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 msgr2=0x7f2adc121110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 0x7f2adc121110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00f770 tx=0x7f2acc00fb30 comp rx=0 tx=0).stop 2024-09-16T11:06:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 msgr2=0x7f2adc121110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:01.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.080+0000 7f2ad2ffd700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc120d30 0x7f2adc121110 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00f770 tx=0x7f2acc00fb30 comp rx=0 tx=0).stop 2024-09-16T11:06:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:06:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 0x7f2adc06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc13d760 con 0x7f2adc121650 2024-09-16T11:06:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 0x7f2adc06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient: get_auth_request con 0x7f2adc121650 auth_method 0 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:01.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 10 monclient(hunting): handle_auth_done global_id 94288 payload 824 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 1 -- 172.21.15.5:0/51573313 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2adc0a3290 con 0x7f2adc121650 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 client.104135.objecter ms_handle_connect 0x7f2adc121650 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 client.104135.objecter resend_mon_ops 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ada59c700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 0x7f2adc06fa50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0015a0 tx=0x7f2acc001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc023070 con 0x7f2adc121650 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 monclient: dump: 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:01.112 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f2acc003e10 con 0x7f2adc121650 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 msgr2=0x7f2adc06fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 0x7f2adc06fa50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0015a0 tx=0x7f2acc001120 comp rx=0 tx=0).stop 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 msgr2=0x7f2adc06fa50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ad2ffd700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2adc121650 0x7f2adc06fa50 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0015a0 tx=0x7f2acc001120 comp rx=0 tx=0).stop 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:06:01.113 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 60, 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 7, 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:06:01.114 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:06:01.115 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.116 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:06:01.117 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:01.118 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:06:01.119 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:06:01.120 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.121 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:06:01.122 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:01.123 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.124 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:06:01.125 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.126 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:01.127 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:06:01.128 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:01.128 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:01.128 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:01.128 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:06:01.128 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:06:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 10 librados: watch_flush enter 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 10 librados: watch_flush exit 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 20 client.104135.objecter shutdown clearing up homeless session... 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.096+0000 7f2ae0bdb700 10 client.104135.objecter successfully canceled tick 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2abc036f60 msgr2=0x7f2abc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/51573313 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2abc036f60 0x7f2abc039420 secure :-1 s=READY pgs=3900 cs=0 l=1 rev1=1 crypto rx=0x7f2adc122040 tx=0x7f2ac4008030 comp rx=0 tx=0).stop 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 10 monclient: shutdown 2024-09-16T11:06:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 msgr2=0x7f2adc1207f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2adad9d700 1 -- 172.21.15.5:0/51573313 reap_dead start 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400efc0 tx=0x7f2ad4009f90 comp rx=0 tx=0).stop 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 shutdown_connections 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 --2- 172.21.15.5:0/51573313 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f2adc13aeb0 0x7f2adc1207f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 >> 172.21.15.5:0/51573313 conn(0x7f2adc0a4fb0 msgr2=0x7f2adc13b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:01.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 shutdown_connections 2024-09-16T11:06:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 -- 172.21.15.5:0/51573313 wait complete. 2024-09-16T11:06:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:01.100+0000 7f2ae0bdb700 1 librados: shutdown 2024-09-16T11:06:01.133 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:06:01.133 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T11:06:01.134 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T11:06:01.134 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T11:06:01.134 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T11:06:01.134 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T11:06:01.134 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:06:01.167 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T11:06:01.168 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T11:06:01.168 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T11:06:01.168 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T11:06:16.170 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T11:06:16.171 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T11:06:16.171 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T11:06:16.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:06:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: get_monmap_and_config 2024-09-16T11:06:16.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: build_initial_monmap 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: monmap: 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:16.290735+0000 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:16.290735+0000 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 Processor -- start 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 -- start start 2024-09-16T11:06:16.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: init 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c8c09e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-b con 0x7f3c8c09d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-c con 0x7f3c8c09dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c070e40 con 0x7f3c8c09e8e0 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0a01d0 con 0x7f3c8c09dfd0 2024-09-16T11:06:16.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c09fef0 con 0x7f3c8c09d6a0 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91a33700 10 monclient(hunting): authenticate will time out at 1834.867233s 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c8bfff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 msgr2=0x7f3c8c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c8bfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c90a31700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3c8c09e8e0 0x7f3c8c09ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57364/0 (socket says 172.21.15.5:57364) 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 learned_addr learned my addr 172.21.15.5:0/4043531421 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91232700 10 monclient(hunting): get_auth_request con 0x7f3c8c09dfd0 auth_method 0 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:16.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.288+0000 7f3c91232700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:16.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.292+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T11:06:16.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.292+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:16.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.292+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:16.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.292+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:16.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.492+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 msgr2=0x7f3c8c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:16.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.492+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 10 monclient(hunting): get_auth_request con 0x7f3c8c09dfd0 auth_method 0 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:16.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.496+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:06:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.896+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 msgr2=0x7f3c8c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.896+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:06:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 10 monclient(hunting): get_auth_request con 0x7f3c8c09dfd0 auth_method 0 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:16.900+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:06:17.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.700+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 msgr2=0x7f3c8c138b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:17.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.700+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:06:17.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 10 monclient(hunting): get_auth_request con 0x7f3c8c09dfd0 auth_method 0 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:17.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:17.704+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:06:19.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): tick 2024-09-16T11:06:19.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): _check_auth_tickets 2024-09-16T11:06:19.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 monclient(hunting): continuing hunt 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 msgr2=0x7f3c8c138b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c8c09d6a0 0x7f3c8c138b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 msgr2=0x7f3c8c13b0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.143:3300/0 conn(0x7f3c8c09dfd0 0x7f3c8c13b0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 -- 172.21.15.5:0/4043531421 >> v1:172.21.15.5:6789/0 conn(0x7f3c8c09e8e0 legacy=0x7f3c8c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient: _add_conns ranks=[3,1] 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c003760 addr v2:172.21.15.5:3300/0 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c6c0062b0 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 -- 172.21.15.5:0/4043531421 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c09f9f0 con 0x7f3c6c0062b0 2024-09-16T11:06:19.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 1 -- 172.21.15.5:0/4043531421 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0a01d0 con 0x7f3c6c003760 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c8a7fc700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.288+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 10 monclient(hunting): get_auth_request con 0x7f3c6c003760 auth_method 0 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.292+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 10 monclient(hunting): get_auth_request con 0x7f3c6c003760 auth_method 0 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:19.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.496+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T11:06:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 10 monclient(hunting): get_auth_request con 0x7f3c6c003760 auth_method 0 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:19.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:19.900+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T11:06:20.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:20.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 10 monclient(hunting): get_auth_request con 0x7f3c6c003760 auth_method 0 2024-09-16T11:06:20.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:20.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:06:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:06:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:06:20.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:20.704+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T11:06:21.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041359111 0 0) 0x7f3c8c09f9f0 con 0x7f3c6c0062b0 2024-09-16T11:06:21.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): _negotiate 2024-09-16T11:06:21.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): my global_id is 94291 2024-09-16T11:06:21.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c780035d0 con 0x7f3c6c0062b0 2024-09-16T11:06:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502740499 0 0) 0x7f3c780035d0 con 0x7f3c6c0062b0 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 msgr2=0x7f3c6c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 --2- 172.21.15.5:0/4043531421 >> v2:172.21.15.5:3300/0 conn(0x7f3c6c003760 0x7f3c6c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 monclient: found mon.noname-c-legacy 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c00b060 con 0x7f3c6c0062b0 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f3c7c004770 con 0x7f3c6c0062b0 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: dump: 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.217 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 >> v1:172.21.15.143:6789/0 conn(0x7f3c6c0062b0 legacy=0x7f3c6c008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 reap_dead start 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 0x7f3c78003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): picked mon.a con 0x7f3c78003a20 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 0x7f3c78004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): picked mon.b con 0x7f3c78004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 0x7f3c7800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): picked mon.c con 0x7f3c78005280 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c0a01d0 con 0x7f3c78003a20 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7800c1b0 con 0x7f3c78004870 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7800c370 con 0x7f3c78005280 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c91232700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 msgr2=0x7f3c7800ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:21.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c91232700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 0x7f3c7800ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 0x7f3c78003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 0x7f3c78004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): get_auth_request con 0x7f3c78003a20 auth_method 0 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): get_auth_request con 0x7f3c78004870 auth_method 0 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_done global_id 94291 payload 293 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 msgr2=0x7f3c7800ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 0x7f3c7800ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 msgr2=0x7f3c78003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:06:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 0x7f3c78003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 monclient: found mon.b 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 1 -- 172.21.15.5:0/4043531421 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780035d0 con 0x7f3c78004870 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c8bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c91a33700 5 monclient: authenticate success, global_id 94291 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c91a33700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.208+0000 7f3c90a31700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 0x7f3c78003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 0x7f3c78004cf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c003db0 tx=0x7f3c780058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f67943ef4f5ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c004b60 con 0x7f3c78004870 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: dump: 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c004ce0 con 0x7f3c78004870 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 4 set_mon_vals no callback set 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 1 -- 172.21.15.5:0/4043531421 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c006310 con 0x7f3c78004870 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c8b7fe700 10 monclient: dump: 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.224 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.225 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 10 monclient: shutdown 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 -- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 msgr2=0x7f3c78004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 0x7f3c78004cf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c003db0 tx=0x7f3c780058a0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 -- 172.21.15.5:0/4043531421 shutdown_connections 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78005280 0x7f3c7800ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78004870 0x7f3c78004cf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 --2- 172.21.15.5:0/4043531421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78003a20 0x7f3c78003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 -- 172.21.15.5:0/4043531421 >> 172.21.15.5:0/4043531421 conn(0x7f3c8c0a45d0 msgr2=0x7f3c8c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 -- 172.21.15.5:0/4043531421 shutdown_connections 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.212+0000 7f3c91a33700 1 -- 172.21.15.5:0/4043531421 wait complete. 2024-09-16T11:06:21.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient: build_initial_monmap 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient: monmap: 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:21.230 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 librados: starting msgr at 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 librados: starting objecter 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 Processor -- start 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 -- start start 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 librados: setting wanted keys 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 librados: calling monclient init 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient: init 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 0x7f3c8c09b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-c con 0x7f3c78003a20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:21.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-a con 0x7f3c78004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 0x7f3c8c09be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): picked mon.noname-b con 0x7f3c78005280 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c124fd0 con 0x7f3c78004870 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1248e0 con 0x7f3c78005280 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c06c570 con 0x7f3c78003a20 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91a33700 10 monclient(hunting): authenticate will time out at 1839.794231s 2024-09-16T11:06:21.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c90a31700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 msgr2=0x7f3c8c09b410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c90a31700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 0x7f3c8c09b410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54110/0 (socket says 172.21.15.5:54110) 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 -- 172.21.15.5:0/1844231045 learned_addr learned my addr 172.21.15.5:0/1844231045 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): get_auth_request con 0x7f3c78004870 auth_method 0 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 0x7f3c8c09be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): get_auth_request con 0x7f3c78005280 auth_method 0 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient(hunting): handle_auth_done global_id 74598 payload 824 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 -- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 msgr2=0x7f3c8c09b410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 0x7f3c8c09b410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 -- 172.21.15.5:0/1844231045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 msgr2=0x7f3c8c09be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 0x7f3c8c09be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 monclient: found mon.noname-a 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 -- 172.21.15.5:0/1844231045 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c09b0d0 con 0x7f3c78004870 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c91232700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 0x7f3c8c09be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c8bfff700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c011000 tx=0x7f3c7c006980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c897fa700 10 client.?.objecter ms_handle_connect 0x7f3c78004870 2024-09-16T11:06:21.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.216+0000 7f3c897fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c024030 con 0x7f3c78004870 2024-09-16T11:06:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:21.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: dump: 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 5 monclient: authenticate success, global_id 74598 2024-09-16T11:06:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c006e70 con 0x7f3c78004870 2024-09-16T11:06:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _renew_subs 2024-09-16T11:06:21.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c0062f0 con 0x7f3c78004870 2024-09-16T11:06:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c09d060 con 0x7f3c78004870 2024-09-16T11:06:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c8affd700 4 set_mon_vals no callback set 2024-09-16T11:06:21.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 client.74598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c897fa700 10 monclient: dump: 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.241 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _renew_subs 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c1242c0 con 0x7f3c78004870 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 1 librados: init done 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.228+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c125440 con 0x7f3c78004870 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3c7c027070 con 0x7f3c78004870 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 1 --2- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 0x7f3c6c039c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(662..662 src has 395..662) v4 ==== 5775+0+0 (secure 0 0 0) 0x7f3c7c025070 con 0x7f3c78004870 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 10 client.74598.objecter ms_dispatch 0x7f3c8c000fd0 osd_map(662..662 src has 395..662) v4 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 3 client.74598.objecter handle_osd_map got epochs [662,662] > 0 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 3 client.74598.objecter handle_osd_map decoding full epoch 662 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c897fa700 20 client.74598.objecter dump_active .. 0 homeless 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c90a31700 1 --2- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 0x7f3c6c039c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.232+0000 7f3c90a31700 10 monclient: get_auth_request con 0x7f3c6c0377d0 auth_method 0 2024-09-16T11:06:21.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.236+0000 7f3c90a31700 1 --2- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 0x7f3c6c039c90 secure :-1 s=READY pgs=3901 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000c00 tx=0x7f3c74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.236+0000 7f3c897fa700 10 client.74598.objecter ms_handle_connect 0x7f3c6c0377d0 2024-09-16T11:06:21.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.240+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3c7c0299c0 con 0x7f3c78004870 2024-09-16T11:06:21.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.240+0000 7f3c897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.240+0000 7f3c897fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:21.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.476+0000 7f3c91a33700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:06:21.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.476+0000 7f3c91a33700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:06:21.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.476+0000 7f3c91a33700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.476+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3c8c06d560 con 0x7f3c78004870 2024-09-16T11:06:21.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c897fa700 1 -- 172.21.15.5:0/1844231045 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1529 (secure 0 0 0) 0x7f3c7c0293d0 con 0x7f3c78004870 2024-09-16T11:06:21.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:06:21.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c897fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:21.483 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":62,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:06:21.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 10 librados: watch_flush enter 2024-09-16T11:06:21.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 10 librados: watch_flush exit 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 20 client.74598.objecter shutdown clearing up homeless session... 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 10 client.74598.objecter successfully canceled tick 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 msgr2=0x7f3c6c039c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 0x7f3c6c039c90 secure :-1 s=READY pgs=3901 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000c00 tx=0x7f3c74009040 comp rx=0 tx=0).stop 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 10 monclient: shutdown 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 msgr2=0x7f3c8c09b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c011000 tx=0x7f3c7c006980 comp rx=0 tx=0).stop 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 shutdown_connections 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c78005280 0x7f3c8c09be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3c6c0377d0 0x7f3c6c039c90 unknown :-1 s=CLOSED pgs=3901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c78004870 0x7f3c8c09b950 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 --2- 172.21.15.5:0/1844231045 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3c78003a20 0x7f3c8c09b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 >> 172.21.15.5:0/1844231045 conn(0x7f3c8c0a45d0 msgr2=0x7f3c8c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 shutdown_connections 2024-09-16T11:06:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 -- 172.21.15.5:0/1844231045 wait complete. 2024-09-16T11:06:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.480+0000 7f3c91a33700 1 librados: shutdown 2024-09-16T11:06:21.497 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T11:06:21.497 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T11:06:21.497 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 10 monclient: get_monmap_and_config 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 10 monclient: build_initial_monmap 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 10 monclient: monmap: 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:21.607431+0000 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:21.607431+0000 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.609 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:21.610 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:21.610 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:21.610 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:21.610 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:21.610 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 1 Processor -- start 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 1 -- start start 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.604+0000 7fd4fa3d7700 10 monclient: init 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T11:06:21.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4f409ea30 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 0x7fd4f413f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-b con 0x7fd4f409f520 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4f409cb20 0x7fd4f41419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-a con 0x7fd4f409cb20 addr v2:172.21.15.5:3300/0 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f409fb90 con 0x7fd4f409ea30 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f409fdf0 con 0x7fd4f409cb20 2024-09-16T11:06:21.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f409aea0 con 0x7fd4f409f520 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient(hunting): authenticate will time out at 1840.184232s 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4ebfff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 msgr2=0x7fd4f413f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 0x7fd4f413f450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4f409cb20 0x7fd4f41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4f409cb20 0x7fd4f41419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54114/0 (socket says 172.21.15.5:54114) 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 -- 172.21.15.5:0/2403058211 learned_addr learned my addr 172.21.15.5:0/2403058211 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): get_auth_request con 0x7fd4f409cb20 auth_method 0 2024-09-16T11:06:21.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient(hunting): handle_auth_done global_id 74601 payload 274 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 -- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 msgr2=0x7fd4f413f450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 --2- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 0x7fd4f413f450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 -- 172.21.15.5:0/2403058211 >> v1:172.21.15.143:6789/0 conn(0x7fd4f409ea30 legacy=0x7fd4f409ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 monclient: found mon.noname-a 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 -- 172.21.15.5:0/2403058211 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f409e3c0 con 0x7fd4f409cb20 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f8974700 1 --2- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409cb20 0x7fd4f41419a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009e50 tx=0x7fd4ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66da6d73814f70d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 1 -- 172.21.15.5:0/2403058211 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec00e070 con 0x7fd4f409cb20 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: dump: 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.619 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 5 monclient: authenticate success, global_id 74601 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 1 -- 172.21.15.5:0/2403058211 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec00b9c0 con 0x7fd4f409cb20 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 1 -- 172.21.15.5:0/2403058211 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec00bd70 con 0x7fd4f409cb20 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 4 set_mon_vals no callback set 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4f37fe700 10 monclient: dump: 2024-09-16T11:06:21.621 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 10 monclient: shutdown 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409cb20 msgr2=0x7fd4f41419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409cb20 0x7fd4f41419a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009e50 tx=0x7fd4ec00b330 comp rx=0 tx=0).stop 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2403058211 shutdown_connections 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409cb20 0x7fd4f41419a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2403058211 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409f520 0x7fd4f413f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2403058211 >> 172.21.15.5:0/2403058211 conn(0x7fd4f40a0b30 msgr2=0x7fd4f40a0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2403058211 shutdown_connections 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.608+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2403058211 wait complete. 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient: build_initial_monmap 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient: monmap: 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:21.623 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 librados: starting msgr at 2024-09-16T11:06:21.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 librados: starting objecter 2024-09-16T11:06:21.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 Processor -- start 2024-09-16T11:06:21.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 -- start start 2024-09-16T11:06:21.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 librados: setting wanted keys 2024-09-16T11:06:21.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 librados: calling monclient init 2024-09-16T11:06:21.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient: init 2024-09-16T11:06:21.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409cb20 0x7fd4f41398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-c con 0x7fd4f409cb20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-a con 0x7fd4f409ea30 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f409f520 0x7fd4f413efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): picked mon.noname-b con 0x7fd4f409f520 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f41243c0 con 0x7fd4f409ea30 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f4141b00 con 0x7fd4f409f520 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f409b1b0 con 0x7fd4f409cb20 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4fa3d7700 10 monclient(hunting): authenticate will time out at 1840.190371s 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f3fff700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409cb20 msgr2=0x7fd4f41398c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f3fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409cb20 0x7fd4f41398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54126/0 (socket says 172.21.15.5:54126) 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 -- 172.21.15.5:0/2668126484 learned_addr learned my addr 172.21.15.5:0/2668126484 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): get_auth_request con 0x7fd4f409ea30 auth_method 0 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f409f520 0x7fd4f413efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): get_auth_request con 0x7fd4f409f520 auth_method 0 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f8974700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient(hunting): handle_auth_done global_id 74604 payload 824 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409cb20 msgr2=0x7fd4f41398c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fd4f409cb20 0x7fd4f41398c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f409f520 msgr2=0x7fd4f413efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4f409f520 0x7fd4f413efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 monclient: found mon.noname-a 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec009b30 con 0x7fd4f409ea30 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4ebfff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400b780 tx=0x7fd4e400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f17fa700 10 client.?.objecter ms_handle_connect 0x7fd4f409ea30 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.612+0000 7fd4f17fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4003e70 con 0x7fd4f409ea30 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: dump: 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e4005070 con 0x7fd4f409ea30 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 5 monclient: authenticate success, global_id 74604 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 monclient: _renew_subs 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4005420 con 0x7fd4f409ea30 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f2ffd700 4 set_mon_vals no callback set 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f41240e0 con 0x7fd4f409ea30 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: dump: 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.639 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 client.74604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 monclient: _renew_subs 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f4140c30 con 0x7fd4f409ea30 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 1 librados: init done 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 0x7fd4f413f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f4067580 con 0x7fd4f413f520 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 0x7fd4f413f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient: get_auth_request con 0x7fd4f413f520 auth_method 0 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fd4e4005580 con 0x7fd4f409ea30 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 --2- 172.21.15.5:0/2668126484 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd4d4036f10 0x7fd4d40393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_done global_id 74607 payload 824 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(662..662 src has 395..662) v4 ==== 5775+0+0 (secure 0 0 0) 0x7fd4e4047970 con 0x7fd4f409ea30 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 client.74604.objecter ms_dispatch 0x7fd4f4000fd0 osd_map(662..662 src has 395..662) v4 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 3 client.74604.objecter handle_osd_map got epochs [662,662] > 0 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 3 client.74604.objecter handle_osd_map decoding full epoch 662 2024-09-16T11:06:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4f4141ee0 con 0x7fd4f413f520 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 20 client.74604.objecter dump_active .. 0 homeless 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f8974700 1 --2- 172.21.15.5:0/2668126484 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd4d4036f10 0x7fd4d40393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 client.74604.objecter ms_handle_connect 0x7fd4f413f520 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 client.74604.objecter resend_mon_ops 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f8974700 10 monclient: get_auth_request con 0x7fd4d4036f10 auth_method 0 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f3fff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 0x7fd4f413f900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc001000 tx=0x7fd4dc00c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc005c80 con 0x7fd4f413f520 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: dump: 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.642 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 client.74604.objecter ms_handle_connect 0x7fd4d4036f10 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f8974700 1 --2- 172.21.15.5:0/2668126484 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd4d4036f10 0x7fd4d40393d0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00b370 tx=0x7fd4ec009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd4dc005de0 con 0x7fd4f413f520 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:21.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 msgr2=0x7fd4f413f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 0x7fd4f413f900 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc001000 tx=0x7fd4dc00c730 comp rx=0 tx=0).stop 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 msgr2=0x7fd4f413f900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.616+0000 7fd4f17fa700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413f520 0x7fd4f413f900 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 0x7fd4f413b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f409ee10 con 0x7fd4f413fe40 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 0x7fd4f413b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient: get_auth_request con 0x7fd4f413fe40 auth_method 0 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 10 monclient(hunting): handle_auth_done global_id 74610 payload 824 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 1 -- 172.21.15.5:0/2668126484 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd4f4123d80 con 0x7fd4f413fe40 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f3fff700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 0x7fd4f413b430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc013000 tx=0x7fd4dc001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 client.74604.objecter ms_handle_connect 0x7fd4f413fe40 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 client.74604.objecter resend_mon_ops 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc020070 con 0x7fd4f413fe40 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 monclient: dump: 2024-09-16T11:06:21.645 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4768 (secure 0 0 0) 0x7fd4dc001240 con 0x7fd4f413fe40 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 msgr2=0x7fd4f413b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 0x7fd4f413b430 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc013000 tx=0x7fd4dc001150 comp rx=0 tx=0).stop 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 msgr2=0x7fd4f413b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f17fa700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f413fe40 0x7fd4f413b430 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:06:21.647 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 62, 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.648 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:06:21.649 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:06:21.650 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:06:21.651 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.652 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:06:21.653 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.654 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:21.656 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.657 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.658 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.659 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.660 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:06:21.661 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 librados: watch_flush enter 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 librados: watch_flush exit 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 20 client.74604.objecter shutdown clearing up homeless session... 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 client.74604.objecter successfully canceled tick 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd4d4036f10 msgr2=0x7fd4d40393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2668126484 >> v2:172.21.15.143:6812/1098205034 conn(0x7fd4d4036f10 0x7fd4d40393d0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00b370 tx=0x7fd4ec009000 comp rx=0 tx=0).stop 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 10 monclient: shutdown 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4f8974700 1 -- 172.21.15.5:0/2668126484 reap_dead start 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 msgr2=0x7fd4f4139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400b780 tx=0x7fd4e400ba90 comp rx=0 tx=0).stop 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 shutdown_connections 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 --2- 172.21.15.5:0/2668126484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4f409ea30 0x7fd4f4139e00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 >> 172.21.15.5:0/2668126484 conn(0x7fd4f40a0b30 msgr2=0x7fd4f40a2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 shutdown_connections 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 -- 172.21.15.5:0/2668126484 wait complete. 2024-09-16T11:06:21.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.628+0000 7fd4fa3d7700 1 librados: shutdown 2024-09-16T11:06:21.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:06:21.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T11:06:21.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:06:21.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 10 monclient: get_monmap_and_config 2024-09-16T11:06:21.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 10 monclient: build_initial_monmap 2024-09-16T11:06:21.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 10 monclient: monmap: 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:06:21.779027+0000 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:06:21.779027+0000 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:06:21.781 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:06:21.782 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:06:21.782 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 1 Processor -- start 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 1 -- start start 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.776+0000 7fb603878700 10 monclient: init 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc0721d0 addr v2:172.21.15.5:3300/0 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5fc070a80 addr v1:172.21.15.143:6789/0 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5fc070ed0 addr v1:172.21.15.5:6789/0 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0a4440 con 0x7fb5fc070ed0 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0725b0 con 0x7fb5fc070a80 2024-09-16T11:06:21.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc06df60 con 0x7fb5fc0721d0 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient(hunting): authenticate will time out at 1840.356439s 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54152/0 (socket says 172.21.15.5:54152) 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 -- 172.21.15.5:0/864979701 learned_addr learned my addr 172.21.15.5:0/864979701 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): get_auth_request con 0x7fb5fc0721d0 auth_method 0 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601e15700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb5fc070ed0 0x7fb5fc132dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57368/0 (socket says 172.21.15.5:57368) 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient(hunting): handle_auth_done global_id 74613 payload 274 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 -- 172.21.15.5:0/864979701 >> v1:172.21.15.143:6789/0 conn(0x7fb5fc070a80 legacy=0x7fb5fc1326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 -- 172.21.15.5:0/864979701 >> v1:172.21.15.5:6789/0 conn(0x7fb5fc070ed0 legacy=0x7fb5fc132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 monclient: found mon.noname-a 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 -- 172.21.15.5:0/864979701 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc1334d0 con 0x7fb5fc0721d0 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb601614700 1 --2- 172.21.15.5:0/864979701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009e50 tx=0x7fb5ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ca410935bf12b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 1 -- 172.21.15.5:0/864979701 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00e070 con 0x7fb5fc0721d0 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: dump: 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.789 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 5 monclient: authenticate success, global_id 74613 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 1 -- 172.21.15.5:0/864979701 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec00b9c0 con 0x7fb5fc0721d0 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 1 -- 172.21.15.5:0/864979701 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00bd70 con 0x7fb5fc0721d0 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: get_monmap_and_config success 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 4 set_mon_vals no callback set 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb5fbfff700 10 monclient: dump: 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.791 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: shutdown 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- 172.21.15.5:0/864979701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc0721d0 msgr2=0x7fb5fc12f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 --2- 172.21.15.5:0/864979701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009e50 tx=0x7fb5ec00b330 comp rx=0 tx=0).stop 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- 172.21.15.5:0/864979701 shutdown_connections 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 --2- 172.21.15.5:0/864979701 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc0721d0 0x7fb5fc12f7c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- 172.21.15.5:0/864979701 >> 172.21.15.5:0/864979701 conn(0x7fb5fc0a18d0 msgr2=0x7fb5fc0a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- 172.21.15.5:0/864979701 shutdown_connections 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- 172.21.15.5:0/864979701 wait complete. 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: build_initial_monmap 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: monmap: 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:06:21.793 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 librados: starting msgr at 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 librados: starting objecter 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 Processor -- start 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 -- start start 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 librados: setting wanted keys 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 1 librados: calling monclient init 2024-09-16T11:06:21.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.780+0000 7fb603878700 10 monclient: init 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _reopen_session rank -1 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc070a80 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc070ed0 0x7fb5fc099fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): picked mon.noname-b con 0x7fb5fc070ed0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5fc0721d0 0x7fb5fc09a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc0721d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc1227f0 con 0x7fb5fc070a80 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0a0e20 con 0x7fb5fc070ed0 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc071a40 con 0x7fb5fc0721d0 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): _renew_subs 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): authenticate will time out at 1840.360684s 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601e15700 1 -- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5fc0721d0 msgr2=0x7fb5fc09a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.5:3301/0 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54162/0 (socket says 172.21.15.5:54162) 2024-09-16T11:06:21.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 -- 172.21.15.5:0/3270125230 learned_addr learned my addr 172.21.15.5:0/3270125230 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601e15700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5fc0721d0 0x7fb5fc09a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): get_auth_request con 0x7fb5fc070a80 auth_method 0 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc070ed0 0x7fb5fc099fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient(hunting): handle_auth_done global_id 74619 payload 824 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5fc0721d0 msgr2=0x7fb5fc09a4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:06:21.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fb5fc0721d0 0x7fb5fc09a4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc070ed0 msgr2=0x7fb5fc099fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc070ed0 0x7fb5fc099fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient: _finish_hunting 0 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 monclient: found mon.noname-a 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec009b30 con 0x7fb5fc070a80 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient: get_auth_request con 0x7fb5fc070ed0 auth_method 0 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc070ed0 0x7fb5fc099fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb601614700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc06fa40 tx=0x7fb5ec004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 client.?.objecter ms_handle_connect 0x7fb5fc070a80 2024-09-16T11:06:21.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec004810 con 0x7fb5fc070a80 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: dump: 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.803 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: _finish_auth 0 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec004970 con 0x7fb5fc070a80 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 5 monclient: authenticate success, global_id 74619 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _renew_subs 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc122d30 con 0x7fb5fc070a80 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:06:21.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec01c680 con 0x7fb5fc070a80 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 monclient: dump: 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.805 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 client.74619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _renew_subs 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc09b6d0 con 0x7fb5fc070a80 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5fb7fe700 4 set_mon_vals no callback set 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 librados: init done 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 0x7fb5fc1cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc067b10 con 0x7fb5fc1caea0 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 0x7fb5fc1cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fb5ec01c7e0 con 0x7fb5fc070a80 2024-09-16T11:06:21.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient: get_auth_request con 0x7fb5fc1caea0 auth_method 0 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 --2- 172.21.15.5:0/3270125230 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb5e8036f10 0x7fb5e80393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(662..662 src has 395..662) v4 ==== 5775+0+0 (secure 0 0 0) 0x7fb5ec049cf0 con 0x7fb5fc070a80 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 10 client.74619.objecter ms_dispatch 0x7fb5fc000fd0 osd_map(662..662 src has 395..662) v4 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 3 client.74619.objecter handle_osd_map got epochs [662,662] > 0 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 3 client.74619.objecter handle_osd_map decoding full epoch 662 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb5f9ffb700 20 client.74619.objecter dump_active .. 0 homeless 2024-09-16T11:06:21.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.784+0000 7fb600e13700 10 monclient(hunting): handle_auth_done global_id 94306 payload 824 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb600e13700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5fc12ef80 con 0x7fb5fc1caea0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb601e15700 1 --2- 172.21.15.5:0/3270125230 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb5e8036f10 0x7fb5e80393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb601e15700 10 monclient: get_auth_request con 0x7fb5e8036f10 auth_method 0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 client.74619.objecter ms_handle_connect 0x7fb5fc1caea0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 client.74619.objecter resend_mon_ops 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 0x7fb5fc1cb280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc1222f0 tx=0x7fb5f000d850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f000b070 con 0x7fb5fc1caea0 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 monclient: dump: 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.808 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb601e15700 1 --2- 172.21.15.5:0/3270125230 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb5e8036f10 0x7fb5e80393d0 secure :-1 s=READY pgs=3903 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4001120 tx=0x7fb5f4007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 client.74619.objecter ms_handle_connect 0x7fb5e8036f10 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fb5f00054e0 con 0x7fb5fc1caea0 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:06:21.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:06:21.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 msgr2=0x7fb5fc1cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 0x7fb5fc1cb280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc1222f0 tx=0x7fb5f000d850 comp rx=0 tx=0).stop 2024-09-16T11:06:21.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 msgr2=0x7fb5fc1cb280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:21.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.788+0000 7fb5f9ffb700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc1caea0 0x7fb5fc1cb280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb603878700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb603878700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 0x7fb5fc02bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb603878700 10 monclient(hunting): start opening mon connection 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc122bd0 con 0x7fb5fc02b6a0 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 0x7fb5fc02bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient: get_auth_request con 0x7fb5fc02b6a0 auth_method 0 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:06:21.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 10 monclient(hunting): handle_auth_done global_id 94309 payload 824 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 1 -- 172.21.15.5:0/3270125230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb5fc06f270 con 0x7fb5fc02b6a0 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb600e13700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 0x7fb5fc02bab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc121da0 tx=0x7fb5f0001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb5f9ffb700 10 client.74619.objecter ms_handle_connect 0x7fb5fc02b6a0 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.796+0000 7fb5f9ffb700 10 client.74619.objecter resend_mon_ops 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0021070 con 0x7fb5fc02b6a0 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:06:21.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 10 monclient: dump: 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4434 (secure 0 0 0) 0x7fb5f0007680 con 0x7fb5fc02b6a0 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:06:21.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 msgr2=0x7fb5fc02bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 0x7fb5fc02bab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc121da0 tx=0x7fb5f0001080 comp rx=0 tx=0).stop 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 msgr2=0x7fb5fc02bab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb5f9ffb700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5fc02b6a0 0x7fb5fc02bab0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc121da0 tx=0x7fb5f0001080 comp rx=0 tx=0).stop 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 62, 2024-09-16T11:06:21.815 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: 1 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.816 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.817 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:06:21.818 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:06:21.819 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.820 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.821 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.822 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:06:21.823 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.824 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:06:21.825 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:06:21.826 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:06:21.827 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 10 librados: watch_flush enter 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 10 librados: watch_flush exit 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 20 client.74619.objecter shutdown clearing up homeless session... 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 10 client.74619.objecter successfully canceled tick 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb5e8036f10 msgr2=0x7fb5e80393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 --2- 172.21.15.5:0/3270125230 >> v2:172.21.15.143:6812/1098205034 conn(0x7fb5e8036f10 0x7fb5e80393d0 secure :-1 s=READY pgs=3903 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4001120 tx=0x7fb5f4007a50 comp rx=0 tx=0).stop 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 10 monclient: shutdown 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 msgr2=0x7fb5fc099a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb601e15700 1 -- 172.21.15.5:0/3270125230 reap_dead start 2024-09-16T11:06:21.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc06fa40 tx=0x7fb5ec004af0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 shutdown_connections 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 --2- 172.21.15.5:0/3270125230 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5fc070a80 0x7fb5fc099a70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 >> 172.21.15.5:0/3270125230 conn(0x7fb5fc0a18d0 msgr2=0x7fb5fc09f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 shutdown_connections 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 -- 172.21.15.5:0/3270125230 wait complete. 2024-09-16T11:06:21.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:06:21.800+0000 7fb603878700 1 librados: shutdown 2024-09-16T11:06:21.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T11:06:21.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (22138 ms) 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (11809 ms) 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8079 ms) 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T11:07:01.244 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13000 ms) 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7927 ms) 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8070 ms) 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: ok, hit_set contains 252:602f83fe:::foo:head 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9003 ms) 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,0 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first is 1726484718 2024-09-16T11:07:01.245 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,1726484726,1726484727,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,1726484726,1726484727,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484718,1726484720,1726484721,1726484723,1726484724,1726484726,1726484727,0 2024-09-16T11:07:01.246 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: got ls 1726484721,1726484723,1726484724,1726484726,1726484727,1726484729,1726484730,0 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: first now 1726484721, trimmed 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20508 ms) 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: foo0 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T11:07:01.247 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14488 ms) 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (37045 ms) 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21936 ms) 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T11:07:01.248 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5050 ms) 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3025 ms) 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2919 ms) 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T11:07:01.249 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3030 ms) 2024-09-16T11:07:01.250 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (263082 ms total) 2024-09-16T11:07:01.250 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: 2024-09-16T11:07:01.250 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T11:07:01.250 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (953012 ms total) 2024-09-16T11:07:01.250 INFO:tasks.workunit.client.0.smithi143.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T11:07:01.257 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.257 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24612 2024-09-16T11:07:01.257 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24612 2024-09-16T11:07:01.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24418 2024-09-16T11:07:01.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24418 2024-09-16T11:07:01.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.258 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24402 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24402 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24441 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24441 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.259 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24651 2024-09-16T11:07:01.260 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24651 2024-09-16T11:07:01.260 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.260 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24822 2024-09-16T11:07:01.260 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24822 2024-09-16T11:07:01.260 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24561 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24561 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24378 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24378 2024-09-16T11:07:01.261 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.262 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24938 2024-09-16T11:07:01.262 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24938 2024-09-16T11:07:01.262 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.262 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24763 2024-09-16T11:07:01.262 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24763 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24593 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24593 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24855 2024-09-16T11:07:01.263 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24855 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24410 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24410 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24774 2024-09-16T11:07:01.264 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24774 2024-09-16T11:07:01.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24394 2024-09-16T11:07:01.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24394 2024-09-16T11:07:01.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.265 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24716 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24716 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24477 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24477 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.266 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24748 2024-09-16T11:07:01.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24748 2024-09-16T11:07:01.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ for t in "${!pids[@]}" 2024-09-16T11:07:01.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ pid=24578 2024-09-16T11:07:01.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ wait 24578 2024-09-16T11:07:01.267 INFO:tasks.workunit.client.0.smithi143.stderr:+ exit 0 2024-09-16T11:07:01.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ cleanup 2024-09-16T11:07:01.268 INFO:tasks.workunit.client.0.smithi143.stderr:+ pkill -P 24364 2024-09-16T11:07:01.270 INFO:tasks.workunit.client.0.smithi143.stderr:+ true 2024-09-16T11:07:01.270 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:07:01.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:07:01.288 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-16T11:07:01.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T11:07:01.615 DEBUG:teuthology.parallel:result is None 2024-09-16T11:07:01.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:07:01.631 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:07:01.631 DEBUG:teuthology.orchestra.run.smithi143:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T11:07:01.655 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:07:01.655 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T11:07:01.665 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T11:07:51.832 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T11:07:51.832 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T11:07:51.833 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T11:07:51.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T11:07:51.839 INFO:tasks.ceph.mon.c:Started 2024-09-16T11:07:51.839 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T11:07:51.839 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T11:07:51.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T11:08:06.840 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T11:08:06.841 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T11:08:06.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:08:06.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.957+0000 7f9204a6c700 10 monclient: get_monmap_and_config 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.957+0000 7f9204a6c700 10 monclient: build_initial_monmap 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.957+0000 7f9204a6c700 10 monclient: monmap: 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:06.964830+0000 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:06.964830+0000 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:06.967 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 Processor -- start 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- start start 2024-09-16T11:08:06.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient: init 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f920009b3e0 0x7f920009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-a con 0x7f920009b3e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f920009bd00 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f920009c110 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:06.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92001334d0 con 0x7f920009bd00 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920009c560 con 0x7f920009c110 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920009e350 con 0x7f920009b3e0 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient(hunting): authenticate will time out at 1945.541682s 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f920009b3e0 0x7f920009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f920009b3e0 0x7f920009b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59764/0 (socket says 172.21.15.5:59764) 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 -- 172.21.15.5:0/4160634640 learned_addr learned my addr 172.21.15.5:0/4160634640 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): get_auth_request con 0x7f920009b3e0 auth_method 0 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:06.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_done global_id 74622 payload 274 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 -- 172.21.15.5:0/4160634640 >> v1:172.21.15.143:6789/0 conn(0x7f920009c110 legacy=0x7f9200132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 -- 172.21.15.5:0/4160634640 >> v1:172.21.15.5:6789/0 conn(0x7f920009bd00 legacy=0x7f92001326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 monclient: found mon.noname-a 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 -- 172.21.15.5:0/4160634640 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920009ca20 con 0x7f920009b3e0 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91ff7fe700 1 --2- 172.21.15.5:0/4160634640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009b3e0 0x7f920009b7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009940 tx=0x7f91f0009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7e8796a7f84323 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 1 -- 172.21.15.5:0/4160634640 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f000ba20 con 0x7f920009b3e0 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: dump: 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 5 monclient: authenticate success, global_id 74622 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 1 -- 172.21.15.5:0/4160634640 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f0009e90 con 0x7f920009b3e0 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:06.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 4 set_mon_vals no callback set 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 1 -- 172.21.15.5:0/4160634640 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f000bd80 con 0x7f920009b3e0 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f91feffd700 10 monclient: dump: 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 10 monclient: shutdown 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- 172.21.15.5:0/4160634640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009b3e0 msgr2=0x7f920009b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 --2- 172.21.15.5:0/4160634640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009b3e0 0x7f920009b7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009940 tx=0x7f91f0009c50 comp rx=0 tx=0).stop 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- 172.21.15.5:0/4160634640 shutdown_connections 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 --2- 172.21.15.5:0/4160634640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009b3e0 0x7f920009b7c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- 172.21.15.5:0/4160634640 >> 172.21.15.5:0/4160634640 conn(0x7f92000a45d0 msgr2=0x7f92000a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- 172.21.15.5:0/4160634640 shutdown_connections 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.961+0000 7f9204a6c700 1 -- 172.21.15.5:0/4160634640 wait complete. 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: build_initial_monmap 2024-09-16T11:08:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: monmap: 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 librados: starting msgr at 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 librados: starting objecter 2024-09-16T11:08:06.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 Processor -- start 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- start start 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 librados: setting wanted keys 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 librados: calling monclient init 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: init 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-c con 0x7f920009b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920009bd00 0x7f92001238a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-b con 0x7f920009bd00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 0x7f9200123de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): picked mon.noname-a con 0x7f920009c110 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:06.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92000a33b0 con 0x7f920009c110 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920009b0d0 con 0x7f920009bd00 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9200070fe0 con 0x7f920009b3e0 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient(hunting): authenticate will time out at 1945.545575s 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 0x7f9200123de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39940/0 (socket says 172.21.15.5:39940) 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 -- 172.21.15.5:0/2520578028 learned_addr learned my addr 172.21.15.5:0/2520578028 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:06.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): get_auth_request con 0x7f920009b3e0 auth_method 0 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): get_auth_request con 0x7f920009c110 auth_method 0 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920009bd00 0x7f92001238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): get_auth_request con 0x7f920009bd00 auth_method 0 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:06.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient(hunting): handle_auth_done global_id 114101 payload 824 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 -- 172.21.15.5:0/2520578028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920009bd00 msgr2=0x7f92001238a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920009bd00 0x7f92001238a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 -- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 msgr2=0x7f9200123de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 0x7f9200123de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:06.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 monclient: found mon.noname-c 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 -- 172.21.15.5:0/2520578028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0009570 con 0x7f920009b3e0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fffff700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 0x7f9200123de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91ff7fe700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f91f000c010 tx=0x7f91f0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 client.?.objecter ms_handle_connect 0x7f920009b3e0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0004380 con 0x7f920009b3e0 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: dump: 2024-09-16T11:08:06.989 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: _finish_auth 0 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: _check_auth_tickets 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:06.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 5 monclient: authenticate success, global_id 114101 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f001f070 con 0x7f920009b3e0 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _renew_subs 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92001273d0 con 0x7f920009b3e0 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f00036c0 con 0x7f920009b3e0 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fcff9700 10 monclient: dump: 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:06.991 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 client.114101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _renew_subs 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9200125c30 con 0x7f920009b3e0 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f91fe7fc700 4 set_mon_vals no callback set 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.965+0000 7f9204a6c700 1 librados: init done 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f91f0021030 con 0x7f920009b3e0 2024-09-16T11:08:06.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f9204a6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f9204a6c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f9204a6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920009dff0 con 0x7f920009b3e0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 1 --2- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 0x7f91e0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f91f0048e40 con 0x7f920009b3e0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 10 client.114101.objecter ms_dispatch 0x7f9200000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 3 client.114101.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 3 client.114101.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 20 client.114101.objecter dump_active .. 0 homeless 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91f6ffd700 1 --2- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 0x7f91e0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91f6ffd700 10 monclient: get_auth_request con 0x7f91e0036f60 auth_method 0 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91fcff9700 10 client.114101.objecter ms_handle_connect 0x7f91e0036f60 2024-09-16T11:08:06.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.969+0000 7f91f6ffd700 1 --2- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 0x7f91e0039420 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f9200070db0 tx=0x7f91e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:06.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.973+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f91f0023990 con 0x7f920009b3e0 2024-09-16T11:08:06.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.973+0000 7f91fcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:06.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:06.973+0000 7f91fcff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f9204a6c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:08:07.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f9204a6c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f9204a6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f92000008d0 con 0x7f920009b3e0 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f91fcff9700 1 -- 172.21.15.5:0/2520578028 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f91f0015070 con 0x7f920009b3e0 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f91fcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.213+0000 7f91fcff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:07.221 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:08:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 10 librados: watch_flush enter 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 10 librados: watch_flush exit 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 20 client.114101.objecter shutdown clearing up homeless session... 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 10 client.114101.objecter successfully canceled tick 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 msgr2=0x7f91e0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 0x7f91e0039420 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f9200070db0 tx=0x7f91e8009040 comp rx=0 tx=0).stop 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 10 monclient: shutdown 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 msgr2=0x7f9200132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f91f000c010 tx=0x7f91f0004770 comp rx=0 tx=0).stop 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 shutdown_connections 2024-09-16T11:08:07.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> v2:172.21.15.143:6812/1098205034 conn(0x7f91e0036f60 0x7f91e0039420 unknown :-1 s=CLOSED pgs=3905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f920009c110 0x7f9200123de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920009bd00 0x7f92001238a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 --2- 172.21.15.5:0/2520578028 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f920009b3e0 0x7f9200132ed0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 >> 172.21.15.5:0/2520578028 conn(0x7f92000a45d0 msgr2=0x7f920006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 shutdown_connections 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 -- 172.21.15.5:0/2520578028 wait complete. 2024-09-16T11:08:07.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.217+0000 7f9204a6c700 1 librados: shutdown 2024-09-16T11:08:07.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:08:07.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T11:08:07.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: monmap: 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:07.358408+0000 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:07.358408+0000 2024-09-16T11:08:07.363 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 Processor -- start 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 -- start start 2024-09-16T11:08:07.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: init 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff41009e8e0 0x7ff41009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff41009e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff41009f200 0x7ff410139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff41009f200 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff41013a460 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41009fb80 con 0x7ff41013a460 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41009c030 con 0x7ff41009f200 2024-09-16T11:08:07.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41009b970 con 0x7ff41009e8e0 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40f59e700 10 monclient(hunting): authenticate will time out at 1945.934775s 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff41009f200 0x7ff410139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff41009f200 0x7ff410139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59788/0 (socket says 172.21.15.5:59788) 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 -- 172.21.15.5:0/652113553 learned_addr learned my addr 172.21.15.5:0/652113553 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): get_auth_request con 0x7ff41009f200 auth_method 0 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40ed9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff41013a460 0x7ff4101411e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51266/0 (socket says 172.21.15.5:51266) 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 1 --2- 172.21.15.5:0/652113553 >> v2:172.21.15.143:3300/0 conn(0x7ff41009e8e0 0x7ff41009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): get_auth_request con 0x7ff41009e8e0 auth_method 0 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_done global_id 74631 payload 274 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 -- 172.21.15.5:0/652113553 >> v2:172.21.15.143:3300/0 conn(0x7ff41009e8e0 msgr2=0x7ff41009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 --2- 172.21.15.5:0/652113553 >> v2:172.21.15.143:3300/0 conn(0x7ff41009e8e0 0x7ff41009ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 -- 172.21.15.5:0/652113553 >> v1:172.21.15.5:6789/0 conn(0x7ff41013a460 legacy=0x7ff4101411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 -- 172.21.15.5:0/652113553 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41009bc50 con 0x7ff41009f200 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.353+0000 7ff40dd9b700 1 --2- 172.21.15.5:0/652113553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410139f20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4100a1180 tx=0x7ff40800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b46ec692da4bf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 1 -- 172.21.15.5:0/652113553 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800e070 con 0x7ff41009f200 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: dump: 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.368 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 1 -- 172.21.15.5:0/652113553 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800b910 con 0x7ff41009f200 2024-09-16T11:08:07.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 1 -- 172.21.15.5:0/652113553 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800bcc0 con 0x7ff41009f200 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 5 monclient: authenticate success, global_id 74631 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 4 set_mon_vals no callback set 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40d59a700 10 monclient: dump: 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 10 monclient: shutdown 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 -- 172.21.15.5:0/652113553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 msgr2=0x7ff410139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 --2- 172.21.15.5:0/652113553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410139f20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4100a1180 tx=0x7ff40800b330 comp rx=0 tx=0).stop 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 -- 172.21.15.5:0/652113553 shutdown_connections 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 --2- 172.21.15.5:0/652113553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410139f20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 --2- 172.21.15.5:0/652113553 >> v2:172.21.15.143:3300/0 conn(0x7ff41009e8e0 0x7ff41009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 -- 172.21.15.5:0/652113553 >> 172.21.15.5:0/652113553 conn(0x7ff4100a45d0 msgr2=0x7ff4100a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 -- 172.21.15.5:0/652113553 shutdown_connections 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 -- 172.21.15.5:0/652113553 wait complete. 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 10 monclient: monmap: 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:07.380 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 librados: starting msgr at 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 librados: starting objecter 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.357+0000 7ff40f59e700 1 Processor -- start 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- start start 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 librados: setting wanted keys 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 librados: calling monclient init 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: init 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff41009e8e0 0x7ff4101265b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff41009e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410126af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff41009f200 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff41013a460 0x7ff410127030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff41013a460 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff410128b60 con 0x7ff41009f200 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41009a540 con 0x7ff41009e8e0 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff410139f40 con 0x7ff41013a460 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): authenticate will time out at 1945.941210s 2024-09-16T11:08:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410126af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410126af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59790/0 (socket says 172.21.15.5:59790) 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 -- 172.21.15.5:0/3793013733 learned_addr learned my addr 172.21.15.5:0/3793013733 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): get_auth_request con 0x7ff41009f200 auth_method 0 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff41009e8e0 0x7ff4101265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40ed9d700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff41013a460 0x7ff410127030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): get_auth_request con 0x7ff41009e8e0 auth_method 0 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40ed9d700 10 monclient(hunting): get_auth_request con 0x7ff41013a460 auth_method 0 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient(hunting): handle_auth_done global_id 74637 payload 824 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff41013a460 msgr2=0x7ff410127030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ff41013a460 0x7ff410127030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff41009e8e0 msgr2=0x7ff4101265b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff41009e8e0 0x7ff4101265b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff408009b30 con 0x7ff41009f200 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff41009e8e0 0x7ff4101265b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:07.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40dd9b700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410126af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff410075050 tx=0x7ff408004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 client.?.objecter ms_handle_connect 0x7ff41009f200 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4080099b0 con 0x7ff41009f200 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: dump: 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.386 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4080092e0 con 0x7ff41009f200 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 5 monclient: authenticate success, global_id 74637 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _renew_subs 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41009b0d0 con 0x7ff41009f200 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40801c660 con 0x7ff41009f200 2024-09-16T11:08:07.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 10 monclient: dump: 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.388 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40cd99700 4 set_mon_vals no callback set 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 client.74637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _renew_subs 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41012a2f0 con 0x7ff41009f200 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 librados: init done 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 0x7ff410123120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41012a480 con 0x7ff410122d40 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 0x7ff410123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient: get_auth_request con 0x7ff410122d40 auth_method 0 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ff40801c7c0 con 0x7ff41009f200 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.361+0000 7ff40e59c700 10 monclient(hunting): handle_auth_done global_id 74640 payload 824 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff40e59c700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff41009f610 con 0x7ff410122d40 2024-09-16T11:08:07.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 --2- 172.21.15.5:0/3793013733 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3f0036f60 0x7ff3f0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7ff4080494a0 con 0x7ff41009f200 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 0x7ff410123120 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8011830 tx=0x7ff3f8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 client.74637.objecter ms_dispatch 0x7ff410000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 3 client.74637.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 3 client.74637.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 20 client.74637.objecter dump_active .. 0 homeless 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 client.74637.objecter ms_handle_connect 0x7ff410122d40 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 client.74637.objecter resend_mon_ops 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8015870 con 0x7ff410122d40 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff40ed9d700 1 --2- 172.21.15.5:0/3793013733 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3f0036f60 0x7ff3f0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 monclient: dump: 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff40ed9d700 10 monclient: get_auth_request con 0x7ff3f0036f60 auth_method 0 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff3f80159d0 con 0x7ff410122d40 2024-09-16T11:08:07.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 msgr2=0x7ff410123120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 0x7ff410123120 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8011830 tx=0x7ff3f8011bf0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 msgr2=0x7ff410123120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410122d40 0x7ff410123120 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff406ffd700 10 client.74637.objecter ms_handle_connect 0x7ff3f0036f60 2024-09-16T11:08:07.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.365+0000 7ff40ed9d700 1 --2- 172.21.15.5:0/3793013733 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3f0036f60 0x7ff3f0039420 secure :-1 s=READY pgs=3906 cs=0 l=1 rev1=1 crypto rx=0x7ff400000f60 tx=0x7ff400007cb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:08:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40f59e700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 0x7ff4101d9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff410067380 con 0x7ff410123660 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 0x7ff4101d9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient: get_auth_request con 0x7ff410123660 auth_method 0 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 10 monclient(hunting): handle_auth_done global_id 74643 payload 824 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 1 -- 172.21.15.5:0/3793013733 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff41009b3e0 con 0x7ff410123660 2024-09-16T11:08:07.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 10 client.74637.objecter ms_handle_connect 0x7ff410123660 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff40e59c700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 0x7ff4101d9000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff3f801b000 tx=0x7ff3f8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 10 client.74637.objecter resend_mon_ops 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8025070 con 0x7ff410123660 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.373+0000 7ff406ffd700 10 monclient: dump: 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.397 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 <== mon.0 v2:172.21.15.5:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4779 (secure 0 0 0) 0x7ff3f80202e0 con 0x7ff410123660 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 msgr2=0x7ff4101d9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 0x7ff4101d9000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff3f801b000 tx=0x7ff3f8004770 comp rx=0 tx=0).stop 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 msgr2=0x7ff4101d9000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff406ffd700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff410123660 0x7ff4101d9000 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "leader", 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 66, 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:08:07.399 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.400 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.401 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:08:07.402 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.403 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:08:07.404 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:08:07.405 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:08:07.406 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.407 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.408 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.409 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.410 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": [ 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 6 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 4 2024-09-16T11:08:07.411 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": [ 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:08:07.412 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 10 librados: watch_flush enter 2024-09-16T11:08:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 10 librados: watch_flush exit 2024-09-16T11:08:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 20 client.74637.objecter shutdown clearing up homeless session... 2024-09-16T11:08:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 10 client.74637.objecter successfully canceled tick 2024-09-16T11:08:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3f0036f60 msgr2=0x7ff3f0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 --2- 172.21.15.5:0/3793013733 >> v2:172.21.15.143:6812/1098205034 conn(0x7ff3f0036f60 0x7ff3f0039420 secure :-1 s=READY pgs=3906 cs=0 l=1 rev1=1 crypto rx=0x7ff400000f60 tx=0x7ff400007cb0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 10 monclient: shutdown 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 msgr2=0x7ff410126af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 --2- 172.21.15.5:0/3793013733 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff41009f200 0x7ff410126af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff410075050 tx=0x7ff408004af0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40ed9d700 1 -- 172.21.15.5:0/3793013733 reap_dead start 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40ed9d700 1 -- 172.21.15.5:0/3793013733 reap_dead start 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 shutdown_connections 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 >> 172.21.15.5:0/3793013733 conn(0x7ff4100a45d0 msgr2=0x7ff41006e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 shutdown_connections 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 -- 172.21.15.5:0/3793013733 wait complete. 2024-09-16T11:08:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.377+0000 7ff40f59e700 1 librados: shutdown 2024-09-16T11:08:07.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:08:07.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T11:08:07.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: monmap: 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:07.533700+0000 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:07.533700+0000 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:07.536 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 Processor -- start 2024-09-16T11:08:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 -- start start 2024-09-16T11:08:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: init 2024-09-16T11:08:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f882409e8e0 0x7f882409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-a con 0x7f882409e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 0x7f882409f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-b con 0x7f882409d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 0x7f88241336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-c con 0x7f882409dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88240a01d0 con 0x7f882409e8e0 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8824070e40 con 0x7f882409dfd0 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882409f740 con 0x7f882409d6a0 2024-09-16T11:08:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient(hunting): authenticate will time out at 1946.110384s 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 0x7f882409f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f882409e8e0 0x7f882409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 0x7f882409f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39972/0 (socket says 172.21.15.5:39972) 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 1 -- 172.21.15.5:0/3635707032 learned_addr learned my addr 172.21.15.5:0/3635707032 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): get_auth_request con 0x7f882409d6a0 auth_method 0 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): get_auth_request con 0x7f882409e8e0 auth_method 0 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 1 --2- 172.21.15.5:0/3635707032 >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 0x7f88241336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 10 monclient(hunting): get_auth_request con 0x7f882409dfd0 auth_method 0 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient(hunting): handle_auth_done global_id 74646 payload 274 2024-09-16T11:08:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 -- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 msgr2=0x7f882409f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 --2- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 0x7f882409f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 -- 172.21.15.5:0/3635707032 >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 msgr2=0x7f88241336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 --2- 172.21.15.5:0/3635707032 >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 0x7f88241336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 -- 172.21.15.5:0/3635707032 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882409fef0 con 0x7f882409e8e0 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f88237fe700 1 --2- 172.21.15.5:0/3635707032 >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 0x7f88241336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8822ffd700 1 --2- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409e8e0 0x7f882409ecc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8824071dd0 tx=0x7f8814009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ee3fb95f7e2999 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 1 -- 172.21.15.5:0/3635707032 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814011830 con 0x7f882409e8e0 2024-09-16T11:08:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: dump: 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 1 -- 172.21.15.5:0/3635707032 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8814011990 con 0x7f882409e8e0 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 5 monclient: authenticate success, global_id 74646 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 1 -- 172.21.15.5:0/3635707032 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814011d20 con 0x7f882409e8e0 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8823fff700 4 set_mon_vals no callback set 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.529+0000 7f8821ffb700 10 monclient: dump: 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 10 monclient: shutdown 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409e8e0 msgr2=0x7f882409ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 --2- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409e8e0 0x7f882409ecc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8824071dd0 tx=0x7f8814009b30 comp rx=0 tx=0).stop 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- 172.21.15.5:0/3635707032 shutdown_connections 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 --2- 172.21.15.5:0/3635707032 >> v2:172.21.15.143:3300/0 conn(0x7f882409dfd0 0x7f88241336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 --2- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409d6a0 0x7f882409f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 --2- 172.21.15.5:0/3635707032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409e8e0 0x7f882409ecc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- 172.21.15.5:0/3635707032 >> 172.21.15.5:0/3635707032 conn(0x7f88240a45d0 msgr2=0x7f88240a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- 172.21.15.5:0/3635707032 shutdown_connections 2024-09-16T11:08:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- 172.21.15.5:0/3635707032 wait complete. 2024-09-16T11:08:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 10 monclient: monmap: 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 librados: starting msgr at 2024-09-16T11:08:07.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 librados: starting objecter 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 Processor -- start 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 -- start start 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 librados: setting wanted keys 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 1 librados: calling monclient init 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.533+0000 7f8823fff700 10 monclient: init 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 0x7f88241c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-a con 0x7f882409d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f882409dfd0 0x7f8824131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-b con 0x7f882409dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:07.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 0x7f8824132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): picked mon.noname-c con 0x7f882409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88241c9420 con 0x7f882409d6a0 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882411dba0 con 0x7f882409dfd0 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882411de20 con 0x7f882409e8e0 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): authenticate will time out at 1946.116364s 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 0x7f88241c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 0x7f8824132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 0x7f88241c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59828/0 (socket says 172.21.15.5:59828) 2024-09-16T11:08:07.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 0x7f8824132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:39978/0 (socket says 172.21.15.5:39978) 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 1 -- 172.21.15.5:0/1921266003 learned_addr learned my addr 172.21.15.5:0/1921266003 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): get_auth_request con 0x7f882409e8e0 auth_method 0 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): get_auth_request con 0x7f882409d6a0 auth_method 0 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f882409dfd0 0x7f8824131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): get_auth_request con 0x7f882409dfd0 auth_method 0 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient(hunting): handle_auth_done global_id 114110 payload 824 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f882409dfd0 msgr2=0x7f8824131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f882409dfd0 0x7f8824131bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 msgr2=0x7f88241c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 0x7f88241c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 monclient: found mon.noname-c 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140094b0 con 0x7f882409e8e0 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f882409d6a0 0x7f88241c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88237fe700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 0x7f8824132130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f881c007ec0 tx=0x7f881c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 client.?.objecter ms_handle_connect 0x7f882409e8e0 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c00dde0 con 0x7f882409e8e0 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: dump: 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 5 monclient: authenticate success, global_id 114110 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _renew_subs 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88240a32a0 con 0x7f882409e8e0 2024-09-16T11:08:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 client.114110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _renew_subs 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882411f2d0 con 0x7f882409e8e0 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 librados: init done 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c00f040 con 0x7f882409e8e0 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c003c40 con 0x7f882409e8e0 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 monclient: dump: 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.565 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f88217fa700 4 set_mon_vals no callback set 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f881c004030 con 0x7f882409e8e0 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 0x7f8808004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8808007d30 con 0x7f8808002850 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 --2- 172.21.15.5:0/1921266003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8804036f60 0x7f8804039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 0x7f8808004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f881c047a90 con 0x7f882409e8e0 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 10 client.114110.objecter ms_dispatch 0x7f8824000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 3 client.114110.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f881bfff700 3 client.114110.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient: get_auth_request con 0x7f8808002850 auth_method 0 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.537+0000 7f8822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 20 client.114110.objecter dump_active .. 0 homeless 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f88227fc700 1 --2- 172.21.15.5:0/1921266003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8804036f60 0x7f8804039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f88227fc700 10 monclient: get_auth_request con 0x7f8804036f60 auth_method 0 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 10 monclient(hunting): handle_auth_done global_id 114113 payload 824 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8808007ef0 con 0x7f8808002850 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 client.114110.objecter ms_handle_connect 0x7f8808002850 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f8822ffd700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 0x7f8808004cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8814009b70 tx=0x7f8814004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 client.114110.objecter resend_mon_ops 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814020070 con 0x7f8808002850 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 monclient: dump: 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 client.114110.objecter ms_handle_connect 0x7f8804036f60 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f88227fc700 1 --2- 172.21.15.5:0/1921266003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8804036f60 0x7f8804039420 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 crypto rx=0x7f882411ddb0 tx=0x7f880c0092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f88140036c0 con 0x7f8808002850 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 msgr2=0x7f8808004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 0x7f8808004cf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8814009b70 tx=0x7f8814004770 comp rx=0 tx=0).stop 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 msgr2=0x7f8808004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.541+0000 7f881bfff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808002850 0x7f8808004cf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8823fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8823fff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 0x7f88080071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8823fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880800a1f0 con 0x7f8808006d70 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 0x7f88080071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient: get_auth_request con 0x7f8808006d70 auth_method 0 2024-09-16T11:08:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.549+0000 7f8822ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8822ffd700 10 monclient(hunting): handle_auth_done global_id 114116 payload 824 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8822ffd700 1 -- 172.21.15.5:0/1921266003 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f880800a3b0 con 0x7f8808006d70 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8822ffd700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 0x7f88080071e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8814023000 tx=0x7f8814011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 client.114110.objecter ms_handle_connect 0x7f8808006d70 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 client.114110.objecter resend_mon_ops 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881401b840 con 0x7f8808006d70 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:07.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 monclient: dump: 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 <== mon.2 v2:172.21.15.5:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f881402c310 con 0x7f8808006d70 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:08:07.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 msgr2=0x7f88080071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 0x7f88080071e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8814023000 tx=0x7f8814011000 comp rx=0 tx=0).stop 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 msgr2=0x7f88080071e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f881bfff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f8808006d70 0x7f88080071e0 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8814023000 tx=0x7f8814011000 comp rx=0 tx=0).stop 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 66, 2024-09-16T11:08:07.575 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.576 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.577 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:08:07.578 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.579 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:08:07.580 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.581 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.583 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.584 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.585 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.586 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 2 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:08:07.587 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 10 librados: watch_flush enter 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 10 librados: watch_flush exit 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 20 client.114110.objecter shutdown clearing up homeless session... 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 10 client.114110.objecter successfully canceled tick 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8804036f60 msgr2=0x7f8804039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 --2- 172.21.15.5:0/1921266003 >> v2:172.21.15.143:6812/1098205034 conn(0x7f8804036f60 0x7f8804039420 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 crypto rx=0x7f882411ddb0 tx=0x7f880c0092e0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 10 monclient: shutdown 2024-09-16T11:08:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 msgr2=0x7f8824132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 --2- 172.21.15.5:0/1921266003 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f882409e8e0 0x7f8824132130 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f881c007ec0 tx=0x7f881c00d3b0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f88237fe700 1 -- 172.21.15.5:0/1921266003 reap_dead start 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f88237fe700 1 -- 172.21.15.5:0/1921266003 reap_dead start 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 shutdown_connections 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 >> 172.21.15.5:0/1921266003 conn(0x7f88240a45d0 msgr2=0x7f88240a1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 shutdown_connections 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 -- 172.21.15.5:0/1921266003 wait complete. 2024-09-16T11:08:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.553+0000 7f8823fff700 1 librados: shutdown 2024-09-16T11:08:07.590 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:08:07.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T11:08:07.591 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.705+0000 7fa020b5e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.705+0000 7fa020b5e700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.705+0000 7fa020b5e700 10 monclient: monmap: 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:07.711681+0000 2024-09-16T11:08:07.713 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:07.711681+0000 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:07.714 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 Processor -- start 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- start start 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient: init 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 0x7fa01c0a1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-b con 0x7fa01c0a0db0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa01c0a16d0 0x7fa01c0a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-a con 0x7fa01c0a16d0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa01c136060 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa01c06e2d0 con 0x7fa01c136060 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01c06da60 con 0x7fa01c0a16d0 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0a1ff0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient(hunting): authenticate will time out at 1946.289130s 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 0x7fa01c136440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51268/0 (socket says 172.21.15.5:51268) 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 -- 172.21.15.5:0/631833259 learned_addr learned my addr 172.21.15.5:0/631833259 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:07.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 --2- 172.21.15.5:0/631833259 >> v2:172.21.15.5:3300/0 conn(0x7fa01c0a16d0 0x7fa01c0a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 0x7fa01c0a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): get_auth_request con 0x7fa01c0a16d0 auth_method 0 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): get_auth_request con 0x7fa01c0a0db0 auth_method 0 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 -- 172.21.15.5:0/631833259 >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 legacy=0x7fa01c136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 -- 172.21.15.5:0/631833259 >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 legacy=0x7fa01c136440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 --1- 172.21.15.5:0/631833259 >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 0x7fa01c136440 :-1 s=OPENED pgs=269 cs=1 l=1).handle_message read tag failed 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01ad9d700 1 --1- 172.21.15.5:0/631833259 >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 0x7fa01c136440 :-1 s=OPENED pgs=269 cs=1 l=1).fault on lossy channel, failing 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.5:6789/0 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient(hunting): handle_auth_done global_id 74652 payload 274 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 -- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 msgr2=0x7fa01c0a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:07.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 0x7fa01c0a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 -- 172.21.15.5:0/631833259 >> v1:172.21.15.5:6789/0 conn(0x7fa01c136060 legacy=0x7fa01c136440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 -- 172.21.15.5:0/631833259 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c0a27c0 con 0x7fa01c0a16d0 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa01a59c700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 0x7fa01c0a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa013fff700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a16d0 0x7fa01c0a1ab0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0723e0 tx=0x7fa004009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503f9e95408bb0a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 1 -- 172.21.15.5:0/631833259 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004004230 con 0x7fa01c0a16d0 2024-09-16T11:08:07.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: dump: 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.726 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 5 monclient: authenticate success, global_id 74652 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 1 -- 172.21.15.5:0/631833259 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004004390 con 0x7fa01c0a16d0 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:07.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 1 -- 172.21.15.5:0/631833259 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0040036c0 con 0x7fa01c0a16d0 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 4 set_mon_vals no callback set 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa019d9b700 10 monclient: dump: 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 10 monclient: shutdown 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a16d0 msgr2=0x7fa01c0a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a16d0 0x7fa01c0a1ab0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0723e0 tx=0x7fa004009ab0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- 172.21.15.5:0/631833259 shutdown_connections 2024-09-16T11:08:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a16d0 0x7fa01c0a1ab0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 --2- 172.21.15.5:0/631833259 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a0db0 0x7fa01c0a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- 172.21.15.5:0/631833259 >> 172.21.15.5:0/631833259 conn(0x7fa01c0a64b0 msgr2=0x7fa01c0a68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- 172.21.15.5:0/631833259 shutdown_connections 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.709+0000 7fa020b5e700 1 -- 172.21.15.5:0/631833259 wait complete. 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient: monmap: 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:07.734 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 librados: starting msgr at 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 librados: starting objecter 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 Processor -- start 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 -- start start 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 librados: setting wanted keys 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 librados: calling monclient init 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient: init 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 0x7fa01c1ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-a con 0x7fa01c0a0db0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:07.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 0x7fa01c1354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-c con 0x7fa01c0a16d0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa01c136060 0x7fa01c135a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): picked mon.noname-b con 0x7fa01c136060 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c1cad90 con 0x7fa01c0a0db0 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c0708f0 con 0x7fa01c136060 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01c09d6a0 con 0x7fa01c0a16d0 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa020b5e700 10 monclient(hunting): authenticate will time out at 1946.293609s 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 0x7fa01c1ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa013fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 0x7fa01c1354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 0x7fa01c1ca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59838/0 (socket says 172.21.15.5:59838) 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 -- 172.21.15.5:0/1565563330 learned_addr learned my addr 172.21.15.5:0/1565563330 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): get_auth_request con 0x7fa01c0a0db0 auth_method 0 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa013fff700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 msgr2=0x7fa01c1354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa013fff700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 msgr2=0x7fa01c1354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 0x7fa01c1354c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 0x7fa01c1354c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient(hunting): handle_auth_done global_id 74655 payload 824 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 msgr2=0x7fa01c1354c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fa01c0a16d0 0x7fa01c1354c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa01c136060 msgr2=0x7fa01c135a00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa01c136060 0x7fa01c135a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c0094b0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa01a59c700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 0x7fa01c1ca850 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00ec10 tx=0x7fa00c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa012ffd700 10 client.?.objecter ms_handle_connect 0x7fa01c0a0db0 2024-09-16T11:08:07.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.713+0000 7fa012ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c00c940 con 0x7fa01c0a0db0 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: dump: 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.744 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 5 monclient: authenticate success, global_id 74655 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 monclient: _renew_subs 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0040094b0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 client.74655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 monclient: _renew_subs 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa01c122ba0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 1 librados: init done 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c00caa0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c00ce10 con 0x7fa01c0a0db0 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa01959a700 4 set_mon_vals no callback set 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 monclient: dump: 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.746 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 0x7fa000004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa000007d30 con 0x7fa000002850 2024-09-16T11:08:07.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fa00c005a40 con 0x7fa01c0a0db0 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 0x7fa000004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient: get_auth_request con 0x7fa000002850 auth_method 0 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 --2- 172.21.15.5:0/1565563330 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9ffc036f60 0x7f9ffc039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7fa00c047eb0 con 0x7fa01c0a0db0 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 10 client.74655.objecter ms_dispatch 0x7fa01c000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 3 client.74655.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:07.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.717+0000 7fa012ffd700 3 client.74655.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 20 client.74655.objecter dump_active .. 0 homeless 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa01ad9d700 1 --2- 172.21.15.5:0/1565563330 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9ffc036f60 0x7f9ffc039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa01ad9d700 10 monclient: get_auth_request con 0x7f9ffc036f60 auth_method 0 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa013fff700 10 monclient(hunting): handle_auth_done global_id 94330 payload 824 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa013fff700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa000007ef0 con 0x7fa000002850 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 0x7fa000004cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa004004700 tx=0x7fa004004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 client.74655.objecter ms_handle_connect 0x7fa000002850 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 client.74655.objecter resend_mon_ops 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 client.74655.objecter ms_handle_connect 0x7f9ffc036f60 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa01ad9d700 1 --2- 172.21.15.5:0/1565563330 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9ffc036f60 0x7f9ffc039420 secure :-1 s=READY pgs=3908 cs=0 l=1 rev1=1 crypto rx=0x7fa014005cb0 tx=0x7fa014001630 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004006ea0 con 0x7fa000002850 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 monclient: dump: 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fa00401f070 con 0x7fa000002850 2024-09-16T11:08:07.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 msgr2=0x7fa000004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 0x7fa000004cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa004004700 tx=0x7fa004004770 comp rx=0 tx=0).stop 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 msgr2=0x7fa000004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.721+0000 7fa012ffd700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000002850 0x7fa000004cf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 0x7fa0000071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000a1f0 con 0x7fa000006d70 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 0x7fa0000071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient: get_auth_request con 0x7fa000006d70 auth_method 0 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 10 monclient(hunting): handle_auth_done global_id 94333 payload 824 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 1 -- 172.21.15.5:0/1565563330 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa00000a3b0 con 0x7fa000006d70 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 client.74655.objecter ms_handle_connect 0x7fa000006d70 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 client.74655.objecter resend_mon_ops 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa013fff700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 0x7fa0000071e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa000007ec0 tx=0x7fa004011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00402b780 con 0x7fa000006d70 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:08:07.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 monclient: dump: 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7fa00402b8e0 con 0x7fa000006d70 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:07.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 msgr2=0x7fa0000071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 0x7fa0000071e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa000007ec0 tx=0x7fa004011000 comp rx=0 tx=0).stop 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 msgr2=0x7fa0000071e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa012ffd700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa000006d70 0x7fa0000071e0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa000007ec0 tx=0x7fa004011000 comp rx=0 tx=0).stop 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout: "state": "peon", 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout: "election_epoch": 66, 2024-09-16T11:08:07.754 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum": [ 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: 0, 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: 1, 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: 2 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_age": 0, 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "required_con": "2449958758054445060", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "required_mon": [ 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.755 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_con": "4540138322906710015", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "quorum_mon": [ 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.756 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "outside_quorum": [], 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "extra_probe_peers": [], 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_provider": [], 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": { 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0ea84356-fa95-422b-b956-38c4a6b7c43e", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "modified": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.757 INFO:teuthology.orchestra.run.smithi005.stdout: "created": "2024-09-16T10:49:21.278459Z", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release": 18, 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "min_mon_release_name": "reef", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "election_strategy": 3, 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "disallowed_leaders: ": "", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false, 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "tiebreaker_mon": "", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "removed_ranks: ": "", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "features": { 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "persistent": [ 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "kraken", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "luminous", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "mimic", 2024-09-16T11:08:07.758 INFO:teuthology.orchestra.run.smithi005.stdout: "osdmap-prune", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "nautilus", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "octopus", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "pacific", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "elector-pinging", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "quincy", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "reef" 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "optional": [] 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "mons": [ 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 0, 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "a", 2024-09-16T11:08:07.759 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3300", 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789", 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.760 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6789/0", 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 1, 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "b", 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.761 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:3300", 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789", 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.762 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "rank": 2, 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "c", 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrs": { 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "addrvec": [ 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v2", 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:3301", 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.763 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "type": "v1", 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790", 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "nonce": 0 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "172.21.15.5:6790/0", 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "priority": 0, 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "weight": 0, 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "{}" 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.764 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "feature_map": { 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": [ 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 1 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "client": [ 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "release": "reef", 2024-09-16T11:08:07.765 INFO:teuthology.orchestra.run.smithi005.stdout: "num": 3 2024-09-16T11:08:07.766 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-09-16T11:08:07.766 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-09-16T11:08:07.766 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-09-16T11:08:07.766 INFO:teuthology.orchestra.run.smithi005.stdout: "stretch_mode": false 2024-09-16T11:08:07.766 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:07.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 librados: watch_flush enter 2024-09-16T11:08:07.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 librados: watch_flush exit 2024-09-16T11:08:07.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 20 client.74655.objecter shutdown clearing up homeless session... 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 client.74655.objecter successfully canceled tick 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9ffc036f60 msgr2=0x7f9ffc039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 --2- 172.21.15.5:0/1565563330 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9ffc036f60 0x7f9ffc039420 secure :-1 s=READY pgs=3908 cs=0 l=1 rev1=1 crypto rx=0x7fa014005cb0 tx=0x7fa014001630 comp rx=0 tx=0).stop 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 10 monclient: shutdown 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 msgr2=0x7fa01c1ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 --2- 172.21.15.5:0/1565563330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0a0db0 0x7fa01c1ca850 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00ec10 tx=0x7fa00c009ed0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa01ad9d700 1 -- 172.21.15.5:0/1565563330 reap_dead start 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa01ad9d700 1 -- 172.21.15.5:0/1565563330 reap_dead start 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 shutdown_connections 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 >> 172.21.15.5:0/1565563330 conn(0x7fa01c0a64b0 msgr2=0x7fa01c09c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 shutdown_connections 2024-09-16T11:08:07.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 -- 172.21.15.5:0/1565563330 wait complete. 2024-09-16T11:08:07.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.733+0000 7fa020b5e700 1 librados: shutdown 2024-09-16T11:08:07.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T11:08:07.769 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T11:08:07.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: get_monmap_and_config 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: monmap: 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:07.885779+0000 2024-09-16T11:08:07.887 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:07.885779+0000 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:07.888 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 Processor -- start 2024-09-16T11:08:07.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 -- start start 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: init 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc04009e8e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc04009d6a0 0x7fc040138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-c con 0x7fc04009d6a0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc04009dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc040070e40 con 0x7fc04009e8e0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04009fef0 con 0x7fc04009dfd0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0400a01d0 con 0x7fc04009d6a0 2024-09-16T11:08:07.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient(hunting): authenticate will time out at 1946.462089s 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0448e5700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc04009e8e0 0x7fc04009ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51282/0 (socket says 172.21.15.5:51282) 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0448e5700 1 -- 172.21.15.5:0/4284910330 learned_addr learned my addr 172.21.15.5:0/4284910330 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 --2- 172.21.15.5:0/4284910330 >> v2:172.21.15.143:3300/0 conn(0x7fc04009d6a0 0x7fc040138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): get_auth_request con 0x7fc04009d6a0 auth_method 0 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient(hunting): handle_auth_done global_id 94336 payload 274 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 -- 172.21.15.5:0/4284910330 >> v1:172.21.15.143:6789/0 conn(0x7fc04009dfd0 legacy=0x7fc04013b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 -- 172.21.15.5:0/4284910330 >> v1:172.21.15.5:6789/0 conn(0x7fc04009e8e0 legacy=0x7fc04009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 monclient: found mon.noname-c 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 1 -- 172.21.15.5:0/4284910330 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371033593 0 0) 0x7fc040070e40 con 0x7fc04009e8e0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 -- 172.21.15.5:0/4284910330 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04009f9f0 con 0x7fc04009d6a0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03ffff700 1 --2- 172.21.15.5:0/4284910330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009d6a0 0x7fc040138b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009880 tx=0x7fc02c009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10a96412ff2cf93c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 1 -- 172.21.15.5:0/4284910330 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c011830 con 0x7fc04009d6a0 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: dump: 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 5 monclient: authenticate success, global_id 94336 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 1 -- 172.21.15.5:0/4284910330 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c011990 con 0x7fc04009d6a0 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc0458e7700 4 set_mon_vals no callback set 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 1 -- 172.21.15.5:0/4284910330 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c011d20 con 0x7fc04009d6a0 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.881+0000 7fc03f7fe700 10 monclient: dump: 2024-09-16T11:08:07.896 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: shutdown 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- 172.21.15.5:0/4284910330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009d6a0 msgr2=0x7fc040138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 --2- 172.21.15.5:0/4284910330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009d6a0 0x7fc040138b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009880 tx=0x7fc02c009b90 comp rx=0 tx=0).stop 2024-09-16T11:08:07.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- 172.21.15.5:0/4284910330 shutdown_connections 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 --2- 172.21.15.5:0/4284910330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009d6a0 0x7fc040138b80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- 172.21.15.5:0/4284910330 >> 172.21.15.5:0/4284910330 conn(0x7fc0400a45d0 msgr2=0x7fc0400a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- 172.21.15.5:0/4284910330 shutdown_connections 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- 172.21.15.5:0/4284910330 wait complete. 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: build_initial_monmap 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: monmap: 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:07.898 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 librados: starting msgr at 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 librados: starting objecter 2024-09-16T11:08:07.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 Processor -- start 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- start start 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 librados: setting wanted keys 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 librados: calling monclient init 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: init 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-c con 0x7fc04009d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009dfd0 0x7fc040132e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-b con 0x7fc04009dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:07.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): picked mon.noname-a con 0x7fc04009e8e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04009b0d0 con 0x7fc04009e8e0 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040120840 con 0x7fc04009dfd0 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04011ffa0 con 0x7fc04009d6a0 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0458e7700 10 monclient(hunting): authenticate will time out at 1946.467313s 2024-09-16T11:08:07.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59852/0 (socket says 172.21.15.5:59852) 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 1 -- 172.21.15.5:0/878776693 learned_addr learned my addr 172.21.15.5:0/878776693 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40018/0 (socket says 172.21.15.5:40018) 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): get_auth_request con 0x7fc04009e8e0 auth_method 0 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): get_auth_request con 0x7fc04009d6a0 auth_method 0 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009dfd0 0x7fc040132e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0448e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): get_auth_request con 0x7fc04009dfd0 auth_method 0 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc0450e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.885+0000 7fc03ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:07.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 10 monclient(hunting): handle_auth_done global_id 74661 payload 824 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 -- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 msgr2=0x7fc0401ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 -- 172.21.15.5:0/878776693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009dfd0 msgr2=0x7fc040132e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009dfd0 0x7fc040132e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 10 monclient: _finish_hunting 0 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 monclient: found mon.noname-a 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 -- 172.21.15.5:0/878776693 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0094b0 con 0x7fc04009e8e0 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0448e5700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0450e6700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc03800cbc0 tx=0x7fc03800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 client.?.objecter ms_handle_connect 0x7fc04009e8e0 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038003c20 con 0x7fc04009e8e0 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: dump: 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: _finish_auth 0 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: _check_auth_tickets 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc038003d80 con 0x7fc04009e8e0 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 5 monclient: authenticate success, global_id 74661 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _renew_subs 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc040070260 con 0x7fc04009e8e0 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03800aae0 con 0x7fc04009e8e0 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:07.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03effd700 4 set_mon_vals no callback set 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 client.74661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 monclient: dump: 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:07.912 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _renew_subs 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0401333c0 con 0x7fc04009e8e0 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 1 librados: init done 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fc03800ac40 con 0x7fc04009e8e0 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 --2- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 0x7fc024039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7fc038047b50 con 0x7fc04009e8e0 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 client.74661.objecter ms_dispatch 0x7fc040000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 3 client.74661.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 3 client.74661.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 20 client.74661.objecter dump_active .. 0 homeless 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04009ba70 con 0x7fc04009e8e0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0448e5700 1 --2- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 0x7fc024039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0448e5700 10 monclient: get_auth_request con 0x7fc024036f60 auth_method 0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc03d7fa700 10 client.74661.objecter ms_handle_connect 0x7fc024036f60 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.889+0000 7fc0448e5700 1 --2- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 0x7fc024039420 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7fc034007b30 tx=0x7fc034007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.897+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc04009ba70 con 0x7fc04009e8e0 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.897+0000 7fc03d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:07.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:07.897+0000 7fc03d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:08.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.133+0000 7fc0458e7700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T11:08:08.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.133+0000 7fc0458e7700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T11:08:08.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.133+0000 7fc0458e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:08.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.133+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc04009ba70 con 0x7fc04009e8e0 2024-09-16T11:08:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc03d7fa700 1 -- 172.21.15.5:0/878776693 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc04009ba70 con 0x7fc04009e8e0 2024-09-16T11:08:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc03d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T11:08:08.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc03d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=10,logm=59} crc {auth=308242996,config=692619927,health=1943650319,kv=564032378,logm=1968025928}) 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=510771600,config=692619927,health=3529651788,kv=564032378,logm=2828816261}) 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=10,logm=59} crc {auth=308242996,config=692619927,health=1943650319,kv=564032378,logm=1968025928}) 2024-09-16T11:08:08.145 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.137+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=1,health=12,kv=9,logm=59} crc {auth=3144496223,config=3678701417,health=1594825231,kv=3001707193,logm=3359486775}) 2024-09-16T11:08:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc0458e7700 10 librados: watch_flush enter 2024-09-16T11:08:08.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc0458e7700 10 librados: watch_flush exit 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc0458e7700 20 client.74661.objecter shutdown clearing up homeless session... 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc0458e7700 10 client.74661.objecter successfully canceled tick 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.137+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 msgr2=0x7fc024039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 0x7fc024039420 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7fc034007b30 tx=0x7fc034007a80 comp rx=0 tx=0).stop 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 10 monclient: shutdown 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 msgr2=0x7fc04006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc03800cbc0 tx=0x7fc03800ced0 comp rx=0 tx=0).stop 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 shutdown_connections 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> v2:172.21.15.143:6812/1098205034 conn(0x7fc024036f60 0x7fc024039420 unknown :-1 s=CLOSED pgs=3909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc04009e8e0 0x7fc04006e650 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc04009dfd0 0x7fc040132e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:08.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 --2- 172.21.15.5:0/878776693 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fc04009d6a0 0x7fc0401ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 >> 172.21.15.5:0/878776693 conn(0x7fc0400a45d0 msgr2=0x7fc040133f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 shutdown_connections 2024-09-16T11:08:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 -- 172.21.15.5:0/878776693 wait complete. 2024-09-16T11:08:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:08.141+0000 7fc0458e7700 1 librados: shutdown 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284491861}) 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501151564}) 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284491861}) 2024-09-16T11:08:08.149 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.141+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187391105}) 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085359124}) 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190837594}) 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085359124}) 2024-09-16T11:08:08.151 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232800280}) 2024-09-16T11:08:08.153 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896393113}) 2024-09-16T11:08:08.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787334218}) 2024-09-16T11:08:08.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896393113}) 2024-09-16T11:08:08.154 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.145+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553047032}) 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785130040}) 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1270203705}) 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785130040}) 2024-09-16T11:08:08.155 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559515331}) 2024-09-16T11:08:08.156 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.156 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501979737}) 2024-09-16T11:08:08.156 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022015062}) 2024-09-16T11:08:08.156 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501979737}) 2024-09-16T11:08:08.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883515370}) 2024-09-16T11:08:08.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791348770}) 2024-09-16T11:08:08.157 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885182105}) 2024-09-16T11:08:08.158 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.158 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791348770}) 2024-09-16T11:08:08.158 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.149+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897824459}) 2024-09-16T11:08:08.158 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.158 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613280309}) 2024-09-16T11:08:08.159 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3118713833}) 2024-09-16T11:08:08.159 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.159 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613280309}) 2024-09-16T11:08:08.159 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606812999}) 2024-09-16T11:08:08.159 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.160 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231880984}) 2024-09-16T11:08:08.160 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59781828}) 2024-09-16T11:08:08.160 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.160 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231880984}) 2024-09-16T11:08:08.160 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577632718}) 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484389427}) 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4257499114}) 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484389427}) 2024-09-16T11:08:08.161 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724837347}) 2024-09-16T11:08:08.162 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T11:08:08.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.153+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054464791}) 2024-09-16T11:08:08.162 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991789995}) 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054464791}) 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152689248}) 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339020535}) 2024-09-16T11:08:08.163 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=161299173}) 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339020535}) 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309034191}) 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409508249}) 2024-09-16T11:08:08.164 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3383088531}) 2024-09-16T11:08:08.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409508249}) 2024-09-16T11:08:08.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102228687}) 2024-09-16T11:08:08.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.165 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642635979}) 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542227648}) 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642635979}) 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.157+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981850974}) 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869739574}) 2024-09-16T11:08:08.166 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037838200}) 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869739574}) 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425463343}) 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582869254}) 2024-09-16T11:08:08.167 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786548482}) 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582869254}) 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3533467636}) 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233415333}) 2024-09-16T11:08:08.168 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3207169438}) 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233415333}) 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923128308}) 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321466611}) 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257447105}) 2024-09-16T11:08:08.169 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321466611}) 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.161+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517523288}) 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=353456196}) 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1592781081}) 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.170 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=353456196}) 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756707244}) 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786525347}) 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387638814}) 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.171 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786525347}) 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565810945}) 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303189253}) 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356477340}) 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.172 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303189253}) 2024-09-16T11:08:08.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220040131}) 2024-09-16T11:08:08.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191878643}) 2024-09-16T11:08:08.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=710503084}) 2024-09-16T11:08:08.173 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191878643}) 2024-09-16T11:08:08.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.165+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=336616106}) 2024-09-16T11:08:08.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.174 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736967422}) 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934456557}) 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736967422}) 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442885213}) 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393030367}) 2024-09-16T11:08:08.175 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365098382}) 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393030367}) 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769905309}) 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844115742}) 2024-09-16T11:08:08.176 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=561382970}) 2024-09-16T11:08:08.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844115742}) 2024-09-16T11:08:08.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390380462}) 2024-09-16T11:08:08.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.177 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778055770}) 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048973856}) 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778055770}) 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.169+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3283027818}) 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.178 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109547238}) 2024-09-16T11:08:08.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639144830}) 2024-09-16T11:08:08.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109547238}) 2024-09-16T11:08:08.179 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3719126855}) 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502622157}) 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838399952}) 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502622157}) 2024-09-16T11:08:08.180 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716284072}) 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504012369}) 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239787970}) 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504012369}) 2024-09-16T11:08:08.181 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704171137}) 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960987396}) 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666801979}) 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960987396}) 2024-09-16T11:08:08.182 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.173+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4010948690}) 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386627029}) 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731656211}) 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386627029}) 2024-09-16T11:08:08.183 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=463913616}) 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667026786}) 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551911288}) 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667026786}) 2024-09-16T11:08:08.184 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550920458}) 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107490521}) 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626859960}) 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107490521}) 2024-09-16T11:08:08.185 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568636336}) 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175731573}) 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208516927}) 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175731573}) 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.177+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369309454}) 2024-09-16T11:08:08.186 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749150493}) 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675850129}) 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749150493}) 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635039931}) 2024-09-16T11:08:08.187 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530893532}) 2024-09-16T11:08:08.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387932604}) 2024-09-16T11:08:08.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1530893532}) 2024-09-16T11:08:08.188 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=795209516}) 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860966984}) 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877655725}) 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860966984}) 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046766947}) 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.189 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941927658}) 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1227958171}) 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941927658}) 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.181+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943388944}) 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.190 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328260728}) 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4001740845}) 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328260728}) 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275113978}) 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659104722}) 2024-09-16T11:08:08.191 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584265655}) 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659104722}) 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903135791}) 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652820591}) 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269747355}) 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.192 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652820591}) 2024-09-16T11:08:08.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021713362}) 2024-09-16T11:08:08.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113995467}) 2024-09-16T11:08:08.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239295663}) 2024-09-16T11:08:08.193 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113995467}) 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.185+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81064188}) 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205180219}) 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3100976894}) 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.194 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205180219}) 2024-09-16T11:08:08.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061002946}) 2024-09-16T11:08:08.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249935817}) 2024-09-16T11:08:08.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511233798}) 2024-09-16T11:08:08.195 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249935817}) 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654561947}) 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865602086}) 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548458250}) 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.196 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865602086}) 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219371743}) 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905274325}) 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613782281}) 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2905274325}) 2024-09-16T11:08:08.197 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.189+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=291643990}) 2024-09-16T11:08:08.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352636507}) 2024-09-16T11:08:08.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857776200}) 2024-09-16T11:08:08.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.198 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1352636507}) 2024-09-16T11:08:08.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266576282}) 2024-09-16T11:08:08.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782429951}) 2024-09-16T11:08:08.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026159877}) 2024-09-16T11:08:08.199 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782429951}) 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549597094}) 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209143077}) 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692075057}) 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.200 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209143077}) 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077339322}) 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473537278}) 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148820396}) 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473537278}) 2024-09-16T11:08:08.201 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.193+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2612219}) 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428602692}) 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182369879}) 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1428602692}) 2024-09-16T11:08:08.202 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501733920}) 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423101821}) 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004908839}) 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423101821}) 2024-09-16T11:08:08.203 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925544513}) 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629704903}) 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14647857}) 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629704903}) 2024-09-16T11:08:08.204 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121070939}) 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620672209}) 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604397284}) 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620672209}) 2024-09-16T11:08:08.205 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.197+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094398226}) 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379842487}) 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879984478}) 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379842487}) 2024-09-16T11:08:08.206 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643519510}) 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899523625}) 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963366025}) 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899523625}) 2024-09-16T11:08:08.207 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304906612}) 2024-09-16T11:08:08.208 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.208 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569595111}) 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2558545041}) 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569595111}) 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783082155}) 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.209 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745078919}) 2024-09-16T11:08:08.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316464174}) 2024-09-16T11:08:08.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745078919}) 2024-09-16T11:08:08.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.201+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1590870077}) 2024-09-16T11:08:08.210 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182031148}) 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=310143561}) 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3182031148}) 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218970723}) 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.211 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106996196}) 2024-09-16T11:08:08.212 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623546648}) 2024-09-16T11:08:08.212 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.212 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106996196}) 2024-09-16T11:08:08.212 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981287963}) 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134824865}) 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328072915}) 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134824865}) 2024-09-16T11:08:08.213 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403192002}) 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247485203}) 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869295309}) 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247485203}) 2024-09-16T11:08:08.214 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.205+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815079149}) 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407949025}) 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797966016}) 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407949025}) 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340764859}) 2024-09-16T11:08:08.215 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310331619}) 2024-09-16T11:08:08.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849901110}) 2024-09-16T11:08:08.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310331619}) 2024-09-16T11:08:08.216 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187785242}) 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113432042}) 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3472067820}) 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3113432042}) 2024-09-16T11:08:08.217 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265767171}) 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643358458}) 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1807647771}) 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643358458}) 2024-09-16T11:08:08.218 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.209+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=574283843}) 2024-09-16T11:08:08.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888013227}) 2024-09-16T11:08:08.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341982117}) 2024-09-16T11:08:08.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.221 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888013227}) 2024-09-16T11:08:08.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.213+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269507175}) 2024-09-16T11:08:08.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494644949}) 2024-09-16T11:08:08.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327605609}) 2024-09-16T11:08:08.222 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494644949}) 2024-09-16T11:08:08.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457541466}) 2024-09-16T11:08:08.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.223 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107980828}) 2024-09-16T11:08:08.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109745333}) 2024-09-16T11:08:08.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107980828}) 2024-09-16T11:08:08.224 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943981965}) 2024-09-16T11:08:08.225 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480776020}) 2024-09-16T11:08:08.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604483977}) 2024-09-16T11:08:08.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480776020}) 2024-09-16T11:08:08.226 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.217+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654409522}) 2024-09-16T11:08:08.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3646131637,mdsmap=1531151340,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1484308106,mon_config_key=4062742737,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3131149949,osd_snap=3083986502}) 2024-09-16T11:08:08.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=1544103285,mdsmap=1531151339,mgr=3082099499,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=2785255227,mon_config_key=4284816287,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3131149949,osd_snap=2561163171}) 2024-09-16T11:08:08.228 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3646131637,mdsmap=1531151340,mgr=3015988023,mgr_command_descs=794974419,mgr_metadata=800353851,mgrstat=1484308106,mon_config_key=4062742737,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3131149949,osd_snap=3083986502}) 2024-09-16T11:08:08.229 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.221+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=10,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=519317227,mdsmap=1531151339,mgr=3903130773,mgr_command_descs=794974419,mgrstat=3918001637,mon_config_key=151770788,monmap=2357222013,osd_metadata=3752038628,osd_pg_creating=3131149949,osd_snap=3594749486}) 2024-09-16T11:08:08.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=40,osdmap=60} crc {osd_snap=3396548488,osdmap=3513485159}) 2024-09-16T11:08:08.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=3712211363,osdmap=3499597436}) 2024-09-16T11:08:08.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.230 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=40,osdmap=60} crc {osd_snap=3396548488,osdmap=3513485159}) 2024-09-16T11:08:08.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=41,osdmap=59} crc {osd_snap=2353549332,osdmap=2814792661}) 2024-09-16T11:08:08.231 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3213925830}) 2024-09-16T11:08:08.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1434082447}) 2024-09-16T11:08:08.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3213925830}) 2024-09-16T11:08:08.232 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.225+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3677917801}) 2024-09-16T11:08:08.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1394349715}) 2024-09-16T11:08:08.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1749240434}) 2024-09-16T11:08:08.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.234 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1394349715}) 2024-09-16T11:08:08.235 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1206155033}) 2024-09-16T11:08:08.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1575785167}) 2024-09-16T11:08:08.236 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3717070986}) 2024-09-16T11:08:08.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1575785167}) 2024-09-16T11:08:08.237 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.229+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3075846069}) 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=86} crc {osdmap=737363551}) 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=89} crc {osdmap=44113749}) 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=86} crc {osdmap=737363551}) 2024-09-16T11:08:08.245 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-09-16T11:08:08.237+0000 7f2fd5f26700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=84} crc {osdmap=967264057}) 2024-09-16T11:08:09.163 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T11:08:09.163 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: get_monmap_and_config 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: build_initial_monmap 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: monmap: 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:09.281545+0000 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:09.281545+0000 2024-09-16T11:08:09.283 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:09.284 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 Processor -- start 2024-09-16T11:08:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 -- start start 2024-09-16T11:08:09.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: init 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f459006efe0 0x7f4590131e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): picked mon.noname-a con 0x7f459006efe0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f45900a0ab0 0x7f459012e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): picked mon.noname-c con 0x7f45900a0ab0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f459012e9a0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4590070e60 con 0x7f459012e9a0 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4590070cd0 con 0x7f459006efe0 2024-09-16T11:08:09.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459006f3c0 con 0x7f45900a0ab0 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f4594892700 10 monclient(hunting): authenticate will time out at 1947.859683s 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f459006efe0 0x7f4590131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f459006efe0 0x7f4590131e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59854/0 (socket says 172.21.15.5:59854) 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 1 -- 172.21.15.5:0/1331470418 learned_addr learned my addr 172.21.15.5:0/1331470418 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 10 monclient(hunting): get_auth_request con 0x7f459006efe0 auth_method 0 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.277+0000 7f458ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458e59c700 1 --2- 172.21.15.5:0/1331470418 >> v2:172.21.15.143:3300/0 conn(0x7f45900a0ab0 0x7f459012e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458e59c700 10 monclient(hunting): get_auth_request con 0x7f45900a0ab0 auth_method 0 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient(hunting): handle_auth_done global_id 74664 payload 274 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 -- 172.21.15.5:0/1331470418 >> v2:172.21.15.143:3300/0 conn(0x7f45900a0ab0 msgr2=0x7f459012e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 --2- 172.21.15.5:0/1331470418 >> v2:172.21.15.143:3300/0 conn(0x7f45900a0ab0 0x7f459012e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 -- 172.21.15.5:0/1331470418 >> v1:172.21.15.143:6789/0 conn(0x7f459012e9a0 legacy=0x7f4590138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient: _finish_hunting 0 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 monclient: found mon.noname-a 2024-09-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 -- 172.21.15.5:0/1331470418 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459006e070 con 0x7f459006efe0 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458ed9d700 1 --2- 172.21.15.5:0/1331470418 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459006efe0 0x7f4590131e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f459006e990 tx=0x7f458000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ad274de92e15b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 1 -- 172.21.15.5:0/1331470418 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458000dd90 con 0x7f459006efe0 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: dump: 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: _finish_auth 0 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: _check_auth_tickets 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 5 monclient: authenticate success, global_id 74664 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 1 -- 172.21.15.5:0/1331470418 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580010470 con 0x7f459006efe0 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 1 -- 172.21.15.5:0/1331470418 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580010800 con 0x7f459006efe0 2024-09-16T11:08:09.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 4 set_mon_vals no callback set 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f458dd9b700 10 monclient: dump: 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.295 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 10 monclient: shutdown 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- 172.21.15.5:0/1331470418 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459006efe0 msgr2=0x7f4590131e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 --2- 172.21.15.5:0/1331470418 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459006efe0 0x7f4590131e80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f459006e990 tx=0x7f458000be90 comp rx=0 tx=0).stop 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- 172.21.15.5:0/1331470418 shutdown_connections 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 --2- 172.21.15.5:0/1331470418 >> v2:172.21.15.143:3300/0 conn(0x7f45900a0ab0 0x7f459012e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 --2- 172.21.15.5:0/1331470418 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459006efe0 0x7f4590131e80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- 172.21.15.5:0/1331470418 >> 172.21.15.5:0/1331470418 conn(0x7f45900a4880 msgr2=0x7f45900a4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- 172.21.15.5:0/1331470418 shutdown_connections 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- 172.21.15.5:0/1331470418 wait complete. 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 10 monclient: build_initial_monmap 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 10 monclient: monmap: 2024-09-16T11:08:09.296 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:09.297 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 librados: starting msgr at 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 librados: starting objecter 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 Processor -- start 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 -- start start 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 librados: setting wanted keys 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 1 librados: calling monclient init 2024-09-16T11:08:09.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.281+0000 7f4594892700 10 monclient: init 2024-09-16T11:08:09.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 0x7f45901cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): picked mon.noname-b con 0x7f459006efe0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): picked mon.noname-c con 0x7f45900a0ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): picked mon.noname-a con 0x7f459012e9a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459012b6c0 con 0x7f459012e9a0 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4590131070 con 0x7f459006efe0 2024-09-16T11:08:09.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4590120420 con 0x7f45900a0ab0 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient(hunting): authenticate will time out at 1947.864904s 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 0x7f45901cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40032/0 (socket says 172.21.15.5:40032) 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59864/0 (socket says 172.21.15.5:59864) 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 1 -- 172.21.15.5:0/4025215486 learned_addr learned my addr 172.21.15.5:0/4025215486 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): get_auth_request con 0x7f459006efe0 auth_method 0 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): get_auth_request con 0x7f45900a0ab0 auth_method 0 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): get_auth_request con 0x7f459012e9a0 auth_method 0 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient(hunting): handle_auth_done global_id 74667 payload 824 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 -- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 msgr2=0x7f45901cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 -- 172.21.15.5:0/4025215486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 msgr2=0x7f45901cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 0x7f45901cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient: _finish_hunting 0 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 monclient: found mon.noname-a 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:09.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 -- 172.21.15.5:0/4025215486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458000b810 con 0x7f459012e9a0 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458e59c700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458ed9d700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 0x7f45901cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458f59e700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f458800cb90 tx=0x7f458800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 client.?.objecter ms_handle_connect 0x7f459012e9a0 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4588003c20 con 0x7f459012e9a0 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: dump: 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.307 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 5 monclient: authenticate success, global_id 74667 2024-09-16T11:08:09.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4588003d80 con 0x7f459012e9a0 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458800aa80 con 0x7f459012e9a0 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _renew_subs 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4590131450 con 0x7f459012e9a0 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f45777fe700 10 monclient: dump: 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.309 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f458d59a700 4 set_mon_vals no callback set 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 client.74667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _renew_subs 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4590120700 con 0x7f459012e9a0 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.285+0000 7f4594892700 1 librados: init done 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f458800abe0 con 0x7f459012e9a0 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f4594892700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f4594892700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f4594892700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459006f850 con 0x7f459012e9a0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 1 --2- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 0x7f457c039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f4588047a20 con 0x7f459012e9a0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 10 client.74667.objecter ms_dispatch 0x7f4590000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 3 client.74667.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 3 client.74667.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 20 client.74667.objecter dump_active .. 0 homeless 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f458ed9d700 1 --2- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 0x7f457c039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f458ed9d700 10 monclient: get_auth_request con 0x7f457c036f90 auth_method 0 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f45777fe700 10 client.74667.objecter ms_handle_connect 0x7f457c036f90 2024-09-16T11:08:09.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.289+0000 7f458ed9d700 1 --2- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 0x7f457c039450 secure :-1 s=READY pgs=3910 cs=0 l=1 rev1=1 crypto rx=0x7f458000bfd0 tx=0x7f458001b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.301+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f45880198f0 con 0x7f459012e9a0 2024-09-16T11:08:09.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.301+0000 7f45777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.301+0000 7f45777fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:09.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f4594892700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T11:08:09.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f4594892700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f4594892700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4590070320 con 0x7f459012e9a0 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f45777fe700 1 -- 172.21.15.5:0/4025215486 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f4588019300 con 0x7f459012e9a0 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f45777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.541+0000 7f45777fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:09.549 INFO:teuthology.orchestra.run.smithi005.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","modified":"2024-09-16T10:49:21.278459Z","created":"2024-09-16T10:49:21.278459Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 10 librados: watch_flush enter 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 10 librados: watch_flush exit 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 20 client.74667.objecter shutdown clearing up homeless session... 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 10 client.74667.objecter successfully canceled tick 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 msgr2=0x7f457c039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 0x7f457c039450 secure :-1 s=READY pgs=3910 cs=0 l=1 rev1=1 crypto rx=0x7f458000bfd0 tx=0x7f458001b040 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 10 monclient: shutdown 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 msgr2=0x7f459009e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f458800cb90 tx=0x7f458800cea0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 shutdown_connections 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> v2:172.21.15.143:6812/1098205034 conn(0x7f457c036f90 0x7f457c039450 unknown :-1 s=CLOSED pgs=3910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f459012e9a0 0x7f459009e1e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f45900a0ab0 0x7f45901cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 --2- 172.21.15.5:0/4025215486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f459006efe0 0x7f45901cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 >> 172.21.15.5:0/4025215486 conn(0x7f45900a4880 msgr2=0x7f45900a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 shutdown_connections 2024-09-16T11:08:09.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 -- 172.21.15.5:0/4025215486 wait complete. 2024-09-16T11:08:09.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.545+0000 7f4594892700 1 librados: shutdown 2024-09-16T11:08:09.568 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T11:08:09.569 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T11:08:09.578 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T11:08:09.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: build_initial_monmap 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: monmap: 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.692 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:09.689962+0000 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:09.689962+0000 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 Processor -- start 2024-09-16T11:08:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 -- start start 2024-09-16T11:08:09.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: init 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdbc409e8e0 0x7fdbc409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdbc409e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbc409f200 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 0x7fdbc41421f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdbc413a630 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc409fb80 con 0x7fdbc409f200 2024-09-16T11:08:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc409c030 con 0x7fdbc409e8e0 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc409b970 con 0x7fdbc413a630 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient(hunting): authenticate will time out at 1948.266336s 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 0x7fdbc41421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc1d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdbc409f200 0x7fdbc4139f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51294/0 (socket says 172.21.15.5:51294) 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc1d9b700 1 -- 172.21.15.5:0/1079490119 learned_addr learned my addr 172.21.15.5:0/1079490119 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 1 --2- 172.21.15.5:0/1079490119 >> v2:172.21.15.5:3300/0 conn(0x7fdbc409e8e0 0x7fdbc409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): get_auth_request con 0x7fdbc413a630 auth_method 0 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): get_auth_request con 0x7fdbc409e8e0 auth_method 0 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 1 -- 172.21.15.5:0/1079490119 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133806474 0 0) 0x7fdbc409fb80 con 0x7fdbc409f200 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient(hunting): handle_auth_done global_id 114128 payload 274 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 -- 172.21.15.5:0/1079490119 >> v2:172.21.15.5:3300/0 conn(0x7fdbc409e8e0 msgr2=0x7fdbc409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 --2- 172.21.15.5:0/1079490119 >> v2:172.21.15.5:3300/0 conn(0x7fdbc409e8e0 0x7fdbc409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 -- 172.21.15.5:0/1079490119 >> v1:172.21.15.5:6789/0 conn(0x7fdbc409f200 legacy=0x7fdbc4139f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient: _finish_hunting 0 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 monclient: found mon.noname-b 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 -- 172.21.15.5:0/1079490119 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc409bc50 con 0x7fdbc413a630 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc2d9d700 1 --2- 172.21.15.5:0/1079490119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 0x7fdbc41421f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009850 tx=0x7fdbbc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a422fd2859ab5b1d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 1 -- 172.21.15.5:0/1079490119 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc011830 con 0x7fdbc413a630 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: dump: 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: _finish_auth 0 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: _check_auth_tickets 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 1 -- 172.21.15.5:0/1079490119 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc011990 con 0x7fdbc413a630 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 5 monclient: authenticate success, global_id 114128 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc359e700 4 set_mon_vals no callback set 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 1 -- 172.21.15.5:0/1079490119 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc011d40 con 0x7fdbc413a630 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.685+0000 7fdbc159a700 10 monclient: dump: 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: shutdown 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- 172.21.15.5:0/1079490119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 msgr2=0x7fdbc41421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- 172.21.15.5:0/1079490119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 0x7fdbc41421f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009850 tx=0x7fdbbc009b60 comp rx=0 tx=0).stop 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- 172.21.15.5:0/1079490119 shutdown_connections 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- 172.21.15.5:0/1079490119 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc413a630 0x7fdbc41421f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- 172.21.15.5:0/1079490119 >> v2:172.21.15.5:3300/0 conn(0x7fdbc409e8e0 0x7fdbc409ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- 172.21.15.5:0/1079490119 >> 172.21.15.5:0/1079490119 conn(0x7fdbc40a45d0 msgr2=0x7fdbc40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- 172.21.15.5:0/1079490119 shutdown_connections 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- 172.21.15.5:0/1079490119 wait complete. 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: build_initial_monmap 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: monmap: 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 librados: starting msgr at 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 librados: starting objecter 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 Processor -- start 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- start start 2024-09-16T11:08:09.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 librados: setting wanted keys 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 librados: calling monclient init 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: init 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-c con 0x7fdbc409e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbc409f200 0x7fdbc4139380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-a con 0x7fdbc409f200 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc413a630 0x7fdbc41398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): picked mon.noname-b con 0x7fdbc413a630 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc41ceda0 con 0x7fdbc409f200 2024-09-16T11:08:09.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4139fd0 con 0x7fdbc413a630 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4126780 con 0x7fdbc409e8e0 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc359e700 10 monclient(hunting): authenticate will time out at 1948.271233s 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40046/0 (socket says 172.21.15.5:40046) 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbc409f200 0x7fdbc4139380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 -- 172.21.15.5:0/848950006 learned_addr learned my addr 172.21.15.5:0/848950006 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): get_auth_request con 0x7fdbc409e8e0 auth_method 0 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): get_auth_request con 0x7fdbc409f200 auth_method 0 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc2d9d700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc413a630 0x7fdbc41398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc2d9d700 10 monclient(hunting): get_auth_request con 0x7fdbc413a630 auth_method 0 2024-09-16T11:08:09.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient(hunting): handle_auth_done global_id 114131 payload 824 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 -- 172.21.15.5:0/848950006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc413a630 msgr2=0x7fdbc41398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc413a630 0x7fdbc41398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 -- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbc409f200 msgr2=0x7fdbc4139380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbc409f200 0x7fdbc4139380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 10 monclient: _finish_hunting 0 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.689+0000 7fdbc259c700 1 monclient: found mon.noname-c 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc259c700 1 -- 172.21.15.5:0/848950006 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0094b0 con 0x7fdbc409e8e0 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc259c700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00ea00 tx=0x7fdbac00ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 client.?.objecter ms_handle_connect 0x7fdbc409e8e0 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac009720 con 0x7fdbc409e8e0 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: dump: 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac00c850 con 0x7fdbc409e8e0 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00cbc0 con 0x7fdbc409e8e0 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 5 monclient: authenticate success, global_id 114131 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _renew_subs 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc0d99700 4 set_mon_vals no callback set 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc409a6f0 con 0x7fdbc409e8e0 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 monclient: dump: 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 client.114131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _renew_subs 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc4126d40 con 0x7fdbc409e8e0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 1 librados: init done 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fdbac00cd20 con 0x7fdbc409e8e0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 --2- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 0x7fdba4039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc409ad40 con 0x7fdbc409e8e0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7fdbac047bf0 con 0x7fdbc409e8e0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 client.114131.objecter ms_dispatch 0x7fdbc4000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 3 client.114131.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 3 client.114131.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 20 client.114131.objecter dump_active .. 0 homeless 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc1d9b700 1 --2- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 0x7fdba4039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc1d9b700 10 monclient: get_auth_request con 0x7fdba4036f90 auth_method 0 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbbaffd700 10 client.114131.objecter ms_handle_connect 0x7fdba4036f90 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.693+0000 7fdbc1d9b700 1 --2- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 0x7fdba4039450 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7fdbc409a660 tx=0x7fdbb4001b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.701+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdbac017020 con 0x7fdbc409e8e0 2024-09-16T11:08:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.701+0000 7fdbbaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:09.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.701+0000 7fdbbaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:09.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.877+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbc409db40 con 0x7fdba4036f90 2024-09-16T11:08:09.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.889+0000 7fdbbaffd700 1 -- 172.21.15.5:0/848950006 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133861 (secure 0 0 0) 0x7fdbc409db40 con 0x7fdba4036f90 2024-09-16T11:08:09.897 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 10 librados: watch_flush enter 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 10 librados: watch_flush exit 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 20 client.114131.objecter shutdown clearing up homeless session... 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 10 client.114131.objecter successfully canceled tick 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 msgr2=0x7fdba4039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 0x7fdba4039450 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7fdbc409a660 tx=0x7fdbb4001b70 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 10 monclient: shutdown 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 msgr2=0x7fdbc41ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00ea00 tx=0x7fdbac00ed10 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 shutdown_connections 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc413a630 0x7fdbc41398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> v2:172.21.15.143:6812/1098205034 conn(0x7fdba4036f90 0x7fdba4039450 unknown :-1 s=CLOSED pgs=3911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdbc409f200 0x7fdbc4139380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 --2- 172.21.15.5:0/848950006 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fdbc409e8e0 0x7fdbc41ce860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 >> 172.21.15.5:0/848950006 conn(0x7fdbc40a45d0 msgr2=0x7fdbc40a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 shutdown_connections 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 -- 172.21.15.5:0/848950006 wait complete. 2024-09-16T11:08:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:09.893+0000 7fdbc359e700 1 librados: shutdown 2024-09-16T11:08:09.901 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:09.913 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1234,"stamp":"2024-09-16T11:08:09.666457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":473120,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":444150,"kb_avail":565757920,"statfs":{"total":579820584960,"available":579336110080,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":454810278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003945"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":663,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104625+0000","last_change":"2024-09-16T10:51:10.895648+0000","last_active":"2024-09-16T11:07:01.104625+0000","last_peered":"2024-09-16T11:07:01.104625+0000","last_clean":"2024-09-16T11:07:01.104625+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:07:01.104625+0000","last_undegraded":"2024-09-16T11:07:01.104625+0000","last_fullsized":"2024-09-16T11:07:01.104625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:14.539597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":636,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221962+0000","last_change":"2024-09-16T10:51:10.811863+0000","last_active":"2024-09-16T11:07:00.221962+0000","last_peered":"2024-09-16T11:07:00.221962+0000","last_clean":"2024-09-16T11:07:00.221962+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:07:00.221962+0000","last_undegraded":"2024-09-16T11:07:00.221962+0000","last_fullsized":"2024-09-16T11:07:00.221962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:51.833056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053499+0000","last_change":"2024-09-16T10:51:10.836991+0000","last_active":"2024-09-16T11:07:00.053499+0000","last_peered":"2024-09-16T11:07:00.053499+0000","last_clean":"2024-09-16T11:07:00.053499+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:07:00.053499+0000","last_undegraded":"2024-09-16T11:07:00.053499+0000","last_fullsized":"2024-09-16T11:07:00.053499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:54.839408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.870842+0000","last_change":"2024-09-16T10:51:10.861754+0000","last_active":"2024-09-16T11:07:01.870842+0000","last_peered":"2024-09-16T11:07:01.870842+0000","last_clean":"2024-09-16T11:07:01.870842+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:07:01.870842+0000","last_undegraded":"2024-09-16T11:07:01.870842+0000","last_fullsized":"2024-09-16T11:07:01.870842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:10.098818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054257+0000","last_change":"2024-09-16T10:51:10.888616+0000","last_active":"2024-09-16T11:07:00.054257+0000","last_peered":"2024-09-16T11:07:00.054257+0000","last_clean":"2024-09-16T11:07:00.054257+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:07:00.054257+0000","last_undegraded":"2024-09-16T11:07:00.054257+0000","last_fullsized":"2024-09-16T11:07:00.054257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:47.053233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551276+0000","last_change":"2024-09-16T10:51:10.820735+0000","last_active":"2024-09-16T11:07:00.551276+0000","last_peered":"2024-09-16T11:07:00.551276+0000","last_clean":"2024-09-16T11:07:00.551276+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:07:00.551276+0000","last_undegraded":"2024-09-16T11:07:00.551276+0000","last_fullsized":"2024-09-16T11:07:00.551276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:00.585239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053844+0000","last_change":"2024-09-16T10:51:10.906082+0000","last_active":"2024-09-16T11:07:00.053844+0000","last_peered":"2024-09-16T11:07:00.053844+0000","last_clean":"2024-09-16T11:07:00.053844+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:07:00.053844+0000","last_undegraded":"2024-09-16T11:07:00.053844+0000","last_fullsized":"2024-09-16T11:07:00.053844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:24.987446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051232+0000","last_change":"2024-09-16T10:51:10.825984+0000","last_active":"2024-09-16T11:07:00.051232+0000","last_peered":"2024-09-16T11:07:00.051232+0000","last_clean":"2024-09-16T11:07:00.051232+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:07:00.051232+0000","last_undegraded":"2024-09-16T11:07:00.051232+0000","last_fullsized":"2024-09-16T11:07:00.051232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:31:17.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551587+0000","last_change":"2024-09-16T10:51:10.851590+0000","last_active":"2024-09-16T11:07:00.551587+0000","last_peered":"2024-09-16T11:07:00.551587+0000","last_clean":"2024-09-16T11:07:00.551587+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:07:00.551587+0000","last_undegraded":"2024-09-16T11:07:00.551587+0000","last_fullsized":"2024-09-16T11:07:00.551587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:52.807901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":637,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221819+0000","last_change":"2024-09-16T10:51:10.818746+0000","last_active":"2024-09-16T11:07:00.221819+0000","last_peered":"2024-09-16T11:07:00.221819+0000","last_clean":"2024-09-16T11:07:00.221819+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:07:00.221819+0000","last_undegraded":"2024-09-16T11:07:00.221819+0000","last_fullsized":"2024-09-16T11:07:00.221819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.511253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961018+0000","last_change":"2024-09-16T10:51:10.916719+0000","last_active":"2024-09-16T11:07:01.961018+0000","last_peered":"2024-09-16T11:07:01.961018+0000","last_clean":"2024-09-16T11:07:01.961018+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:07:01.961018+0000","last_undegraded":"2024-09-16T11:07:01.961018+0000","last_fullsized":"2024-09-16T11:07:01.961018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:07.031545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":647,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959927+0000","last_change":"2024-09-16T10:51:10.807192+0000","last_active":"2024-09-16T11:07:01.959927+0000","last_peered":"2024-09-16T11:07:01.959927+0000","last_clean":"2024-09-16T11:07:01.959927+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:07:01.959927+0000","last_undegraded":"2024-09-16T11:07:01.959927+0000","last_fullsized":"2024-09-16T11:07:01.959927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:17.091636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958919+0000","last_change":"2024-09-16T10:51:10.888609+0000","last_active":"2024-09-16T11:07:01.958919+0000","last_peered":"2024-09-16T11:07:01.958919+0000","last_clean":"2024-09-16T11:07:01.958919+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:07:01.958919+0000","last_undegraded":"2024-09-16T11:07:01.958919+0000","last_fullsized":"2024-09-16T11:07:01.958919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:51:59.284621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233208+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T11:07:01.233208+0000","last_peered":"2024-09-16T11:07:01.233208+0000","last_clean":"2024-09-16T11:07:01.233208+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:07:01.233208+0000","last_undegraded":"2024-09-16T11:07:01.233208+0000","last_fullsized":"2024-09-16T11:07:01.233208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961200+0000","last_change":"2024-09-16T10:51:10.907339+0000","last_active":"2024-09-16T11:07:01.961200+0000","last_peered":"2024-09-16T11:07:01.961200+0000","last_clean":"2024-09-16T11:07:01.961200+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:07:01.961200+0000","last_undegraded":"2024-09-16T11:07:01.961200+0000","last_fullsized":"2024-09-16T11:07:01.961200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:22.367946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959643+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T11:07:01.959643+0000","last_peered":"2024-09-16T11:07:01.959643+0000","last_clean":"2024-09-16T11:07:01.959643+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:07:01.959643+0000","last_undegraded":"2024-09-16T11:07:01.959643+0000","last_fullsized":"2024-09-16T11:07:01.959643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":554,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051532+0000","last_change":"2024-09-16T10:52:56.726039+0000","last_active":"2024-09-16T11:07:00.051532+0000","last_peered":"2024-09-16T11:07:00.051532+0000","last_clean":"2024-09-16T11:07:00.051532+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:07:00.051532+0000","last_undegraded":"2024-09-16T11:07:00.051532+0000","last_fullsized":"2024-09-16T11:07:00.051532+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.112159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":648,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221678+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T11:07:00.221678+0000","last_peered":"2024-09-16T11:07:00.221678+0000","last_clean":"2024-09-16T11:07:00.221678+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:07:00.221678+0000","last_undegraded":"2024-09-16T11:07:00.221678+0000","last_fullsized":"2024-09-16T11:07:00.221678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961095+0000","last_change":"2024-09-16T10:51:10.915110+0000","last_active":"2024-09-16T11:07:01.961095+0000","last_peered":"2024-09-16T11:07:01.961095+0000","last_clean":"2024-09-16T11:07:01.961095+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:07:01.961095+0000","last_undegraded":"2024-09-16T11:07:01.961095+0000","last_fullsized":"2024-09-16T11:07:01.961095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:54.203725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":658,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051747+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T11:07:00.051747+0000","last_peered":"2024-09-16T11:07:00.051747+0000","last_clean":"2024-09-16T11:07:00.051747+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:07:00.051747+0000","last_undegraded":"2024-09-16T11:07:00.051747+0000","last_fullsized":"2024-09-16T11:07:00.051747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":677,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104372+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T11:07:01.104372+0000","last_peered":"2024-09-16T11:07:01.104372+0000","last_clean":"2024-09-16T11:07:01.104372+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:07:01.104372+0000","last_undegraded":"2024-09-16T11:07:01.104372+0000","last_fullsized":"2024-09-16T11:07:01.104372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958388+0000","last_change":"2024-09-16T10:51:10.900283+0000","last_active":"2024-09-16T11:07:01.958388+0000","last_peered":"2024-09-16T11:07:01.958388+0000","last_clean":"2024-09-16T11:07:01.958388+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:07:01.958388+0000","last_undegraded":"2024-09-16T11:07:01.958388+0000","last_fullsized":"2024-09-16T11:07:01.958388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:51.095223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":644,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551103+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T11:07:00.551103+0000","last_peered":"2024-09-16T11:07:00.551103+0000","last_clean":"2024-09-16T11:07:00.551103+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:07:00.551103+0000","last_undegraded":"2024-09-16T11:07:00.551103+0000","last_fullsized":"2024-09-16T11:07:00.551103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959213+0000","last_change":"2024-09-16T10:51:10.777356+0000","last_active":"2024-09-16T11:07:01.959213+0000","last_peered":"2024-09-16T11:07:01.959213+0000","last_clean":"2024-09-16T11:07:01.959213+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:07:01.959213+0000","last_undegraded":"2024-09-16T11:07:01.959213+0000","last_fullsized":"2024-09-16T11:07:01.959213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:54.038836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232503+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T11:07:01.232503+0000","last_peered":"2024-09-16T11:07:01.232503+0000","last_clean":"2024-09-16T11:07:01.232503+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:07:01.232503+0000","last_undegraded":"2024-09-16T11:07:01.232503+0000","last_fullsized":"2024-09-16T11:07:01.232503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232530+0000","last_change":"2024-09-16T10:51:10.824359+0000","last_active":"2024-09-16T11:07:01.232530+0000","last_peered":"2024-09-16T11:07:01.232530+0000","last_clean":"2024-09-16T11:07:01.232530+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:07:01.232530+0000","last_undegraded":"2024-09-16T11:07:01.232530+0000","last_fullsized":"2024-09-16T11:07:01.232530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:09.821124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":676,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104414+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T11:07:01.104414+0000","last_peered":"2024-09-16T11:07:01.104414+0000","last_clean":"2024-09-16T11:07:01.104414+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:07:01.104414+0000","last_undegraded":"2024-09-16T11:07:01.104414+0000","last_fullsized":"2024-09-16T11:07:01.104414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":797,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232630+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T11:07:01.232630+0000","last_peered":"2024-09-16T11:07:01.232630+0000","last_clean":"2024-09-16T11:07:01.232630+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:07:01.232630+0000","last_undegraded":"2024-09-16T11:07:01.232630+0000","last_fullsized":"2024-09-16T11:07:01.232630+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232619+0000","last_change":"2024-09-16T10:51:10.819816+0000","last_active":"2024-09-16T11:07:01.232619+0000","last_peered":"2024-09-16T11:07:01.232619+0000","last_clean":"2024-09-16T11:07:01.232619+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:07:01.232619+0000","last_undegraded":"2024-09-16T11:07:01.232619+0000","last_fullsized":"2024-09-16T11:07:01.232619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:01.570513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232841+0000","last_change":"2024-09-16T10:51:10.881698+0000","last_active":"2024-09-16T11:07:01.232841+0000","last_peered":"2024-09-16T11:07:01.232841+0000","last_clean":"2024-09-16T11:07:01.232841+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:07:01.232841+0000","last_undegraded":"2024-09-16T11:07:01.232841+0000","last_fullsized":"2024-09-16T11:07:01.232841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:17.134723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959641+0000","last_change":"2024-09-16T10:51:10.823091+0000","last_active":"2024-09-16T11:07:01.959641+0000","last_peered":"2024-09-16T11:07:01.959641+0000","last_clean":"2024-09-16T11:07:01.959641+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:07:01.959641+0000","last_undegraded":"2024-09-16T11:07:01.959641+0000","last_fullsized":"2024-09-16T11:07:01.959641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:32.535561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551175+0000","last_change":"2024-09-16T10:51:10.890050+0000","last_active":"2024-09-16T11:07:00.551175+0000","last_peered":"2024-09-16T11:07:00.551175+0000","last_clean":"2024-09-16T11:07:00.551175+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:07:00.551175+0000","last_undegraded":"2024-09-16T11:07:00.551175+0000","last_fullsized":"2024-09-16T11:07:00.551175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:00.950193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":649,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054067+0000","last_change":"2024-09-16T10:51:10.851438+0000","last_active":"2024-09-16T11:07:00.054067+0000","last_peered":"2024-09-16T11:07:00.054067+0000","last_clean":"2024-09-16T11:07:00.054067+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:07:00.054067+0000","last_undegraded":"2024-09-16T11:07:00.054067+0000","last_fullsized":"2024-09-16T11:07:00.054067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:26.462001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":539,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871448+0000","last_change":"2024-09-16T10:52:56.729849+0000","last_active":"2024-09-16T11:07:01.871448+0000","last_peered":"2024-09-16T11:07:01.871448+0000","last_clean":"2024-09-16T11:07:01.871448+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:07:01.871448+0000","last_undegraded":"2024-09-16T11:07:01.871448+0000","last_fullsized":"2024-09-16T11:07:01.871448+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:09.227854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":635,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221534+0000","last_change":"2024-09-16T10:51:10.773626+0000","last_active":"2024-09-16T11:07:00.221534+0000","last_peered":"2024-09-16T11:07:00.221534+0000","last_clean":"2024-09-16T11:07:00.221534+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:07:00.221534+0000","last_undegraded":"2024-09-16T11:07:00.221534+0000","last_fullsized":"2024-09-16T11:07:00.221534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:31.797539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":631,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871689+0000","last_change":"2024-09-16T10:51:10.895063+0000","last_active":"2024-09-16T11:07:01.871689+0000","last_peered":"2024-09-16T11:07:01.871689+0000","last_clean":"2024-09-16T11:07:01.871689+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:07:01.871689+0000","last_undegraded":"2024-09-16T11:07:01.871689+0000","last_fullsized":"2024-09-16T11:07:01.871689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:59.114090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":636,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233205+0000","last_change":"2024-09-16T10:51:10.830383+0000","last_active":"2024-09-16T11:07:01.233205+0000","last_peered":"2024-09-16T11:07:01.233205+0000","last_clean":"2024-09-16T11:07:01.233205+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:07:01.233205+0000","last_undegraded":"2024-09-16T11:07:01.233205+0000","last_fullsized":"2024-09-16T11:07:01.233205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:39.706733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":633,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958113+0000","last_change":"2024-09-16T10:51:10.849321+0000","last_active":"2024-09-16T11:07:01.958113+0000","last_peered":"2024-09-16T11:07:01.958113+0000","last_clean":"2024-09-16T11:07:01.958113+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:07:01.958113+0000","last_undegraded":"2024-09-16T11:07:01.958113+0000","last_fullsized":"2024-09-16T11:07:01.958113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:42.862278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"20'7","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959504+0000","last_change":"2024-09-16T10:51:10.808306+0000","last_active":"2024-09-16T11:07:01.959504+0000","last_peered":"2024-09-16T11:07:01.959504+0000","last_clean":"2024-09-16T11:07:01.959504+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:07:01.959504+0000","last_undegraded":"2024-09-16T11:07:01.959504+0000","last_fullsized":"2024-09-16T11:07:01.959504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:18.223681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":664,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.960466+0000","last_change":"2024-09-16T10:51:10.869585+0000","last_active":"2024-09-16T11:07:01.960466+0000","last_peered":"2024-09-16T11:07:01.960466+0000","last_clean":"2024-09-16T11:07:01.960466+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:07:01.960466+0000","last_undegraded":"2024-09-16T11:07:01.960466+0000","last_fullsized":"2024-09-16T11:07:01.960466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:37.869026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232719+0000","last_change":"2024-09-16T10:51:10.814816+0000","last_active":"2024-09-16T11:07:01.232719+0000","last_peered":"2024-09-16T11:07:01.232719+0000","last_clean":"2024-09-16T11:07:01.232719+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:07:01.232719+0000","last_undegraded":"2024-09-16T11:07:01.232719+0000","last_fullsized":"2024-09-16T11:07:01.232719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:29:46.969757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80048,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291792,"statfs":{"total":96636764160,"available":96554795008,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.60299999999999998,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":1.123,"15min":1.984},"last":0.73099999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.80800000000000005},"min":{"1min":0.65800000000000003,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.3},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.74399999999999999,"5min":0.56999999999999995,"15min":0.53000000000000003},"max":{"1min":1.1459999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.75900000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.68100000000000005,"5min":0.54200000000000004,"15min":0.45400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.1779999999999999},"last":0.624},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":1.0620000000000001,"15min":1.22},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67600000000000005,"15min":0.73199999999999998},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.385},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82699999999999996,"15min":0.78900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.365},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.72299999999999998,"15min":0.71199999999999997},"min":{"1min":0.58999999999999997,"5min":0.53800000000000003,"15min":0.39400000000000002},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95399999999999996,"15min":0.92900000000000005},"min":{"1min":0.83399999999999996,"5min":0.72799999999999998,"15min":0.499},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.91800000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80040,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291800,"statfs":{"total":96636764160,"available":96554803200,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.78600000000000003,"15min":0.82899999999999996},"min":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.36599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":5.351},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80000000000000004,"15min":0.78000000000000003},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.29999999999999999},"max":{"1min":1.032,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90800000000000003,"15min":0.91800000000000004},"min":{"1min":0.623,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.268,"5min":1.268,"15min":10.134},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.91900000000000004,"15min":0.91200000000000003},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":0.89300000000000002},"min":{"1min":0.625,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.46899999999999997},"max":{"1min":1.044,"5min":1.28,"15min":1.3859999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.34100000000000003},"max":{"1min":0.86199999999999999,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":0.996,"5min":1.268,"15min":5.0579999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75900000000000001,"15min":0.77000000000000002},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.996,"5min":1.153,"15min":3.7770000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.83499999999999996},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":1.111,"15min":8.5559999999999992},"last":0.68300000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76476,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295364,"statfs":{"total":96636764160,"available":96558452736,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.37},"max":{"1min":0.97799999999999998,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.35599999999999998},"max":{"1min":0.93400000000000005,"5min":1.1000000000000001,"15min":1.609},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.871,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.436},"max":{"1min":1.01,"5min":1.4430000000000001,"15min":1.821},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.85499999999999998,"5min":0.64500000000000002,"15min":0.46899999999999997},"max":{"1min":1.101,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.67700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98899999999999999,"5min":1.2290000000000001,"15min":10.129},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.65400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.006,"5min":2.3559999999999999,"15min":10.449999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":3.4710000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.47999999999999998,"15min":0.33500000000000002},"max":{"1min":0.98399999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.749,"15min":0.77000000000000002},"min":{"1min":0.56200000000000006,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":1.179,"5min":1.1870000000000001,"15min":1.353},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81200000000000006,"15min":0.84199999999999997},"min":{"1min":0.76000000000000001,"5min":0.52200000000000002,"15min":0.39600000000000002},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":2.625},"last":0.75900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71848,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299992,"statfs":{"total":96636764160,"available":96563191808,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.39500000000000002},"max":{"1min":0.94899999999999995,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.65300000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.76900000000000002,"15min":0.746},"min":{"1min":0.63200000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":1.155,"15min":1.155},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0569999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.89400000000000002},"min":{"1min":0.71599999999999997,"5min":0.66100000000000003,"15min":0.60099999999999998},"max":{"1min":1.1739999999999999,"5min":1.341,"15min":1.774},"last":0.97499999999999998}]},{"osd":4,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88700000000000001,"15min":0.89900000000000002},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.59499999999999997},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90100000000000002,"15min":0.875},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59899999999999998},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":1.7090000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.88300000000000001,"15min":0.88100000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.141,"5min":1.397,"15min":3.7989999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.95499999999999996,"15min":0.95099999999999996},"min":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.57799999999999996},"max":{"1min":1.1950000000000001,"5min":1.204,"15min":3.2799999999999998},"last":0.84099999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84144,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":79486,"kb_avail":94287696,"statfs":{"total":96636764160,"available":96550600704,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81394119},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.72699999999999998},"min":{"1min":0.59499999999999997,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":1.04,"15min":1.845},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.51900000000000002,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":1.034,"5min":1.034,"15min":4.9260000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.89700000000000002,"15min":2.8370000000000002},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.879,"15min":0.89200000000000002},"min":{"1min":0.76400000000000001,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.95799999999999996,"15min":0.96799999999999997},"min":{"1min":0.746,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.208,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.89700000000000002},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.93899999999999995,"5min":1.2729999999999999,"15min":2.802},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94899999999999995,"15min":0.94199999999999995},"min":{"1min":0.76700000000000002,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.206,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.90800000000000003},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.71299999999999997,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.159,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.69399999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80564,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291276,"statfs":{"total":96636764160,"available":96554266624,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.221},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":2.6469999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.247},"max":{"1min":1.079,"5min":1.079,"15min":2.3479999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71499999999999997,"5min":1.0289999999999999,"15min":1.327},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":1.228,"15min":4.1319999999999997},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85799999999999998,"15min":0.90300000000000002},"min":{"1min":0.69399999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.70699999999999996,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":1.175,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.88700000000000001}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.88500000000000001,"15min":0.91700000000000004},"min":{"1min":0.73099999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.008,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.74299999999999999,"5min":0.46999999999999997,"15min":0.36699999999999999},"max":{"1min":1.1020000000000001,"5min":1.302,"15min":3.1440000000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.89200000000000002},"min":{"1min":0.69399999999999995,"5min":0.49399999999999999,"15min":0.45200000000000001},"max":{"1min":1.1479999999999999,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88,"15min":0.94499999999999995},"min":{"1min":0.69399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":1.0409999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:09.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:10.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: monmap: 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:10.027696+0000 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:10.027696+0000 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:10.030 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:10.031 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 1 Processor -- start 2024-09-16T11:08:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 1 -- start start 2024-09-16T11:08:10.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: init 2024-09-16T11:08:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.021+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1c6c0a2bf0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c6c09f780 0x7f1c6c0a3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c6c09f780 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c6c1359b0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c0a3a50 con 0x7f1c6c1359b0 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c0a3c40 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c0a4300 con 0x7f1c6c09f780 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient(hunting): authenticate will time out at 1948.604071s 2024-09-16T11:08:10.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59896/0 (socket says 172.21.15.5:59896) 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 -- 172.21.15.5:0/2763525596 learned_addr learned my addr 172.21.15.5:0/2763525596 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): get_auth_request con 0x7f1c6c0a2bf0 auth_method 0 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 1 --2- 172.21.15.5:0/2763525596 >> v2:172.21.15.143:3300/0 conn(0x7f1c6c09f780 0x7f1c6c0a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): get_auth_request con 0x7f1c6c09f780 auth_method 0 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_done global_id 74679 payload 274 2024-09-16T11:08:10.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 -- 172.21.15.5:0/2763525596 >> v2:172.21.15.143:3300/0 conn(0x7f1c6c09f780 msgr2=0x7f1c6c0a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 --2- 172.21.15.5:0/2763525596 >> v2:172.21.15.143:3300/0 conn(0x7f1c6c09f780 0x7f1c6c0a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 -- 172.21.15.5:0/2763525596 >> v1:172.21.15.5:6789/0 conn(0x7f1c6c1359b0 legacy=0x7f1c6c135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 monclient: found mon.noname-a 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 -- 172.21.15.5:0/2763525596 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c09fda0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6a59c700 1 --2- 172.21.15.5:0/2763525596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c0a60a0 tx=0x7f1c6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3350c7952c2f2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 1 -- 172.21.15.5:0/2763525596 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400e070 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: dump: 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.038 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 5 monclient: authenticate success, global_id 74679 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 1 -- 172.21.15.5:0/2763525596 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6400b910 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 4 set_mon_vals no callback set 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 1 -- 172.21.15.5:0/2763525596 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400bcc0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6959a700 10 monclient: dump: 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.040 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 10 monclient: shutdown 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- 172.21.15.5:0/2763525596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c0a2bf0 msgr2=0x7f1c6c0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/2763525596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c0a60a0 tx=0x7f1c6400b330 comp rx=0 tx=0).stop 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- 172.21.15.5:0/2763525596 shutdown_connections 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/2763525596 >> v2:172.21.15.143:3300/0 conn(0x7f1c6c09f780 0x7f1c6c0a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/2763525596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c0a2bf0 0x7f1c6c0a2fd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- 172.21.15.5:0/2763525596 >> 172.21.15.5:0/2763525596 conn(0x7f1c6c0a53e0 msgr2=0x7f1c6c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:10.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- 172.21.15.5:0/2763525596 shutdown_connections 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.025+0000 7f1c6b59e700 1 -- 172.21.15.5:0/2763525596 wait complete. 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: monmap: 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:10.042 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 librados: starting msgr at 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 librados: starting objecter 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 Processor -- start 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 -- start start 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 librados: setting wanted keys 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 librados: calling monclient init 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: init 2024-09-16T11:08:10.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c6c09f780 0x7f1c6c1cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1c6c09f780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c6c0a2bf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c1359b0 0x7f1c6c1210d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1c6c1359b0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c070560 con 0x7f1c6c1359b0 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c1253d0 con 0x7f1c6c09f780 2024-09-16T11:08:10.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c09f4e0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient(hunting): authenticate will time out at 1948.610592s 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c1359b0 0x7f1c6c1210d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40052/0 (socket says 172.21.15.5:40052) 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 -- 172.21.15.5:0/1822334402 learned_addr learned my addr 172.21.15.5:0/1822334402 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): get_auth_request con 0x7f1c6c0a2bf0 auth_method 0 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c6c09f780 0x7f1c6c1cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): get_auth_request con 0x7f1c6c1359b0 auth_method 0 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): get_auth_request con 0x7f1c6c09f780 auth_method 0 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient(hunting): handle_auth_done global_id 114134 payload 824 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 -- 172.21.15.5:0/1822334402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c6c09f780 msgr2=0x7f1c6c1cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c6c09f780 0x7f1c6c1cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 -- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c1359b0 msgr2=0x7f1c6c1210d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c1359b0 0x7f1c6c1210d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 monclient: found mon.noname-c 2024-09-16T11:08:10.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 -- 172.21.15.5:0/1822334402 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c64009b30 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c69d9b700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000e9a0 tx=0x7f1c6000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 client.?.objecter ms_handle_connect 0x7f1c6c0a2bf0 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60009720 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: dump: 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.052 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 5 monclient: authenticate success, global_id 114134 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6000c690 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: _renew_subs 2024-09-16T11:08:10.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c0a66f0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.029+0000 7f1c5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6000ca40 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5bfff700 4 set_mon_vals no callback set 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 10 monclient: dump: 2024-09-16T11:08:10.054 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 client.114134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 monclient: _renew_subs 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c124710 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 1 librados: init done 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1c6000cba0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 1 --2- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 0x7f1c4c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c124b80 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f1c60047fb0 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 10 client.114134.objecter ms_dispatch 0x7f1c6c000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:10.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 3 client.114134.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 3 client.114134.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6a59c700 1 --2- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 0x7f1c4c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6a59c700 10 monclient: get_auth_request con 0x7f1c4c036f60 auth_method 0 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 20 client.114134.objecter dump_active .. 0 homeless 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c5affd700 10 client.114134.objecter ms_handle_connect 0x7f1c4c036f60 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.033+0000 7f1c6a59c700 1 --2- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 0x7f1c4c039420 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7f1c64000f80 tx=0x7f1c64009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.037+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1c60017020 con 0x7f1c6c0a2bf0 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.037+0000 7f1c5affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.037+0000 7f1c5affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:10.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.209+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c6c0a4040 con 0x7f1c4c036f60 2024-09-16T11:08:10.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c5affd700 1 -- 172.21.15.5:0/1822334402 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133861 (secure 0 0 0) 0x7f1c64021070 con 0x7f1c4c036f60 2024-09-16T11:08:10.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 10 librados: watch_flush enter 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 10 librados: watch_flush exit 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 20 client.114134.objecter shutdown clearing up homeless session... 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 10 client.114134.objecter successfully canceled tick 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 msgr2=0x7f1c4c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 0x7f1c4c039420 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7f1c64000f80 tx=0x7f1c64009000 comp rx=0 tx=0).stop 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 10 monclient: shutdown 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 msgr2=0x7f1c6c1cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000e9a0 tx=0x7f1c6000ecb0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 shutdown_connections 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c4c036f60 0x7f1c4c039420 unknown :-1 s=CLOSED pgs=3912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c6c1359b0 0x7f1c6c1210d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c6c0a2bf0 0x7f1c6c1cb390 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 --2- 172.21.15.5:0/1822334402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c6c09f780 0x7f1c6c1cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 >> 172.21.15.5:0/1822334402 conn(0x7f1c6c0a53e0 msgr2=0x7f1c6c135290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 shutdown_connections 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 -- 172.21.15.5:0/1822334402 wait complete. 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.221+0000 7f1c6b59e700 1 librados: shutdown 2024-09-16T11:08:10.230 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:10.242 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1234,"stamp":"2024-09-16T11:08:09.666457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":473120,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":444150,"kb_avail":565757920,"statfs":{"total":579820584960,"available":579336110080,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":454810278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003945"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":663,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104625+0000","last_change":"2024-09-16T10:51:10.895648+0000","last_active":"2024-09-16T11:07:01.104625+0000","last_peered":"2024-09-16T11:07:01.104625+0000","last_clean":"2024-09-16T11:07:01.104625+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:07:01.104625+0000","last_undegraded":"2024-09-16T11:07:01.104625+0000","last_fullsized":"2024-09-16T11:07:01.104625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:14.539597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":636,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221962+0000","last_change":"2024-09-16T10:51:10.811863+0000","last_active":"2024-09-16T11:07:00.221962+0000","last_peered":"2024-09-16T11:07:00.221962+0000","last_clean":"2024-09-16T11:07:00.221962+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:07:00.221962+0000","last_undegraded":"2024-09-16T11:07:00.221962+0000","last_fullsized":"2024-09-16T11:07:00.221962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:51.833056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053499+0000","last_change":"2024-09-16T10:51:10.836991+0000","last_active":"2024-09-16T11:07:00.053499+0000","last_peered":"2024-09-16T11:07:00.053499+0000","last_clean":"2024-09-16T11:07:00.053499+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:07:00.053499+0000","last_undegraded":"2024-09-16T11:07:00.053499+0000","last_fullsized":"2024-09-16T11:07:00.053499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:54.839408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.870842+0000","last_change":"2024-09-16T10:51:10.861754+0000","last_active":"2024-09-16T11:07:01.870842+0000","last_peered":"2024-09-16T11:07:01.870842+0000","last_clean":"2024-09-16T11:07:01.870842+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:07:01.870842+0000","last_undegraded":"2024-09-16T11:07:01.870842+0000","last_fullsized":"2024-09-16T11:07:01.870842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:10.098818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054257+0000","last_change":"2024-09-16T10:51:10.888616+0000","last_active":"2024-09-16T11:07:00.054257+0000","last_peered":"2024-09-16T11:07:00.054257+0000","last_clean":"2024-09-16T11:07:00.054257+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:07:00.054257+0000","last_undegraded":"2024-09-16T11:07:00.054257+0000","last_fullsized":"2024-09-16T11:07:00.054257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:47.053233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551276+0000","last_change":"2024-09-16T10:51:10.820735+0000","last_active":"2024-09-16T11:07:00.551276+0000","last_peered":"2024-09-16T11:07:00.551276+0000","last_clean":"2024-09-16T11:07:00.551276+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:07:00.551276+0000","last_undegraded":"2024-09-16T11:07:00.551276+0000","last_fullsized":"2024-09-16T11:07:00.551276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:00.585239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053844+0000","last_change":"2024-09-16T10:51:10.906082+0000","last_active":"2024-09-16T11:07:00.053844+0000","last_peered":"2024-09-16T11:07:00.053844+0000","last_clean":"2024-09-16T11:07:00.053844+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:07:00.053844+0000","last_undegraded":"2024-09-16T11:07:00.053844+0000","last_fullsized":"2024-09-16T11:07:00.053844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:24.987446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051232+0000","last_change":"2024-09-16T10:51:10.825984+0000","last_active":"2024-09-16T11:07:00.051232+0000","last_peered":"2024-09-16T11:07:00.051232+0000","last_clean":"2024-09-16T11:07:00.051232+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:07:00.051232+0000","last_undegraded":"2024-09-16T11:07:00.051232+0000","last_fullsized":"2024-09-16T11:07:00.051232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:31:17.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551587+0000","last_change":"2024-09-16T10:51:10.851590+0000","last_active":"2024-09-16T11:07:00.551587+0000","last_peered":"2024-09-16T11:07:00.551587+0000","last_clean":"2024-09-16T11:07:00.551587+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:07:00.551587+0000","last_undegraded":"2024-09-16T11:07:00.551587+0000","last_fullsized":"2024-09-16T11:07:00.551587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:52.807901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":637,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221819+0000","last_change":"2024-09-16T10:51:10.818746+0000","last_active":"2024-09-16T11:07:00.221819+0000","last_peered":"2024-09-16T11:07:00.221819+0000","last_clean":"2024-09-16T11:07:00.221819+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:07:00.221819+0000","last_undegraded":"2024-09-16T11:07:00.221819+0000","last_fullsized":"2024-09-16T11:07:00.221819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.511253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961018+0000","last_change":"2024-09-16T10:51:10.916719+0000","last_active":"2024-09-16T11:07:01.961018+0000","last_peered":"2024-09-16T11:07:01.961018+0000","last_clean":"2024-09-16T11:07:01.961018+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:07:01.961018+0000","last_undegraded":"2024-09-16T11:07:01.961018+0000","last_fullsized":"2024-09-16T11:07:01.961018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:07.031545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":647,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959927+0000","last_change":"2024-09-16T10:51:10.807192+0000","last_active":"2024-09-16T11:07:01.959927+0000","last_peered":"2024-09-16T11:07:01.959927+0000","last_clean":"2024-09-16T11:07:01.959927+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:07:01.959927+0000","last_undegraded":"2024-09-16T11:07:01.959927+0000","last_fullsized":"2024-09-16T11:07:01.959927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:17.091636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958919+0000","last_change":"2024-09-16T10:51:10.888609+0000","last_active":"2024-09-16T11:07:01.958919+0000","last_peered":"2024-09-16T11:07:01.958919+0000","last_clean":"2024-09-16T11:07:01.958919+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:07:01.958919+0000","last_undegraded":"2024-09-16T11:07:01.958919+0000","last_fullsized":"2024-09-16T11:07:01.958919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:51:59.284621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233208+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T11:07:01.233208+0000","last_peered":"2024-09-16T11:07:01.233208+0000","last_clean":"2024-09-16T11:07:01.233208+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:07:01.233208+0000","last_undegraded":"2024-09-16T11:07:01.233208+0000","last_fullsized":"2024-09-16T11:07:01.233208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961200+0000","last_change":"2024-09-16T10:51:10.907339+0000","last_active":"2024-09-16T11:07:01.961200+0000","last_peered":"2024-09-16T11:07:01.961200+0000","last_clean":"2024-09-16T11:07:01.961200+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:07:01.961200+0000","last_undegraded":"2024-09-16T11:07:01.961200+0000","last_fullsized":"2024-09-16T11:07:01.961200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:22.367946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959643+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T11:07:01.959643+0000","last_peered":"2024-09-16T11:07:01.959643+0000","last_clean":"2024-09-16T11:07:01.959643+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:07:01.959643+0000","last_undegraded":"2024-09-16T11:07:01.959643+0000","last_fullsized":"2024-09-16T11:07:01.959643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":554,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051532+0000","last_change":"2024-09-16T10:52:56.726039+0000","last_active":"2024-09-16T11:07:00.051532+0000","last_peered":"2024-09-16T11:07:00.051532+0000","last_clean":"2024-09-16T11:07:00.051532+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:07:00.051532+0000","last_undegraded":"2024-09-16T11:07:00.051532+0000","last_fullsized":"2024-09-16T11:07:00.051532+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.112159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":648,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221678+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T11:07:00.221678+0000","last_peered":"2024-09-16T11:07:00.221678+0000","last_clean":"2024-09-16T11:07:00.221678+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:07:00.221678+0000","last_undegraded":"2024-09-16T11:07:00.221678+0000","last_fullsized":"2024-09-16T11:07:00.221678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961095+0000","last_change":"2024-09-16T10:51:10.915110+0000","last_active":"2024-09-16T11:07:01.961095+0000","last_peered":"2024-09-16T11:07:01.961095+0000","last_clean":"2024-09-16T11:07:01.961095+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:07:01.961095+0000","last_undegraded":"2024-09-16T11:07:01.961095+0000","last_fullsized":"2024-09-16T11:07:01.961095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:54.203725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":658,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051747+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T11:07:00.051747+0000","last_peered":"2024-09-16T11:07:00.051747+0000","last_clean":"2024-09-16T11:07:00.051747+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:07:00.051747+0000","last_undegraded":"2024-09-16T11:07:00.051747+0000","last_fullsized":"2024-09-16T11:07:00.051747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":677,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104372+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T11:07:01.104372+0000","last_peered":"2024-09-16T11:07:01.104372+0000","last_clean":"2024-09-16T11:07:01.104372+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:07:01.104372+0000","last_undegraded":"2024-09-16T11:07:01.104372+0000","last_fullsized":"2024-09-16T11:07:01.104372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958388+0000","last_change":"2024-09-16T10:51:10.900283+0000","last_active":"2024-09-16T11:07:01.958388+0000","last_peered":"2024-09-16T11:07:01.958388+0000","last_clean":"2024-09-16T11:07:01.958388+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:07:01.958388+0000","last_undegraded":"2024-09-16T11:07:01.958388+0000","last_fullsized":"2024-09-16T11:07:01.958388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:51.095223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":644,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551103+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T11:07:00.551103+0000","last_peered":"2024-09-16T11:07:00.551103+0000","last_clean":"2024-09-16T11:07:00.551103+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:07:00.551103+0000","last_undegraded":"2024-09-16T11:07:00.551103+0000","last_fullsized":"2024-09-16T11:07:00.551103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959213+0000","last_change":"2024-09-16T10:51:10.777356+0000","last_active":"2024-09-16T11:07:01.959213+0000","last_peered":"2024-09-16T11:07:01.959213+0000","last_clean":"2024-09-16T11:07:01.959213+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:07:01.959213+0000","last_undegraded":"2024-09-16T11:07:01.959213+0000","last_fullsized":"2024-09-16T11:07:01.959213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:54.038836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232503+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T11:07:01.232503+0000","last_peered":"2024-09-16T11:07:01.232503+0000","last_clean":"2024-09-16T11:07:01.232503+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:07:01.232503+0000","last_undegraded":"2024-09-16T11:07:01.232503+0000","last_fullsized":"2024-09-16T11:07:01.232503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232530+0000","last_change":"2024-09-16T10:51:10.824359+0000","last_active":"2024-09-16T11:07:01.232530+0000","last_peered":"2024-09-16T11:07:01.232530+0000","last_clean":"2024-09-16T11:07:01.232530+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:07:01.232530+0000","last_undegraded":"2024-09-16T11:07:01.232530+0000","last_fullsized":"2024-09-16T11:07:01.232530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:09.821124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":676,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104414+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T11:07:01.104414+0000","last_peered":"2024-09-16T11:07:01.104414+0000","last_clean":"2024-09-16T11:07:01.104414+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:07:01.104414+0000","last_undegraded":"2024-09-16T11:07:01.104414+0000","last_fullsized":"2024-09-16T11:07:01.104414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":797,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232630+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T11:07:01.232630+0000","last_peered":"2024-09-16T11:07:01.232630+0000","last_clean":"2024-09-16T11:07:01.232630+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:07:01.232630+0000","last_undegraded":"2024-09-16T11:07:01.232630+0000","last_fullsized":"2024-09-16T11:07:01.232630+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232619+0000","last_change":"2024-09-16T10:51:10.819816+0000","last_active":"2024-09-16T11:07:01.232619+0000","last_peered":"2024-09-16T11:07:01.232619+0000","last_clean":"2024-09-16T11:07:01.232619+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:07:01.232619+0000","last_undegraded":"2024-09-16T11:07:01.232619+0000","last_fullsized":"2024-09-16T11:07:01.232619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:01.570513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232841+0000","last_change":"2024-09-16T10:51:10.881698+0000","last_active":"2024-09-16T11:07:01.232841+0000","last_peered":"2024-09-16T11:07:01.232841+0000","last_clean":"2024-09-16T11:07:01.232841+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:07:01.232841+0000","last_undegraded":"2024-09-16T11:07:01.232841+0000","last_fullsized":"2024-09-16T11:07:01.232841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:17.134723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959641+0000","last_change":"2024-09-16T10:51:10.823091+0000","last_active":"2024-09-16T11:07:01.959641+0000","last_peered":"2024-09-16T11:07:01.959641+0000","last_clean":"2024-09-16T11:07:01.959641+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:07:01.959641+0000","last_undegraded":"2024-09-16T11:07:01.959641+0000","last_fullsized":"2024-09-16T11:07:01.959641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:32.535561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551175+0000","last_change":"2024-09-16T10:51:10.890050+0000","last_active":"2024-09-16T11:07:00.551175+0000","last_peered":"2024-09-16T11:07:00.551175+0000","last_clean":"2024-09-16T11:07:00.551175+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:07:00.551175+0000","last_undegraded":"2024-09-16T11:07:00.551175+0000","last_fullsized":"2024-09-16T11:07:00.551175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:00.950193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":649,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054067+0000","last_change":"2024-09-16T10:51:10.851438+0000","last_active":"2024-09-16T11:07:00.054067+0000","last_peered":"2024-09-16T11:07:00.054067+0000","last_clean":"2024-09-16T11:07:00.054067+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:07:00.054067+0000","last_undegraded":"2024-09-16T11:07:00.054067+0000","last_fullsized":"2024-09-16T11:07:00.054067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:26.462001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":539,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871448+0000","last_change":"2024-09-16T10:52:56.729849+0000","last_active":"2024-09-16T11:07:01.871448+0000","last_peered":"2024-09-16T11:07:01.871448+0000","last_clean":"2024-09-16T11:07:01.871448+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:07:01.871448+0000","last_undegraded":"2024-09-16T11:07:01.871448+0000","last_fullsized":"2024-09-16T11:07:01.871448+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:09.227854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":635,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221534+0000","last_change":"2024-09-16T10:51:10.773626+0000","last_active":"2024-09-16T11:07:00.221534+0000","last_peered":"2024-09-16T11:07:00.221534+0000","last_clean":"2024-09-16T11:07:00.221534+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:07:00.221534+0000","last_undegraded":"2024-09-16T11:07:00.221534+0000","last_fullsized":"2024-09-16T11:07:00.221534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:31.797539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":631,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871689+0000","last_change":"2024-09-16T10:51:10.895063+0000","last_active":"2024-09-16T11:07:01.871689+0000","last_peered":"2024-09-16T11:07:01.871689+0000","last_clean":"2024-09-16T11:07:01.871689+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:07:01.871689+0000","last_undegraded":"2024-09-16T11:07:01.871689+0000","last_fullsized":"2024-09-16T11:07:01.871689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:59.114090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":636,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233205+0000","last_change":"2024-09-16T10:51:10.830383+0000","last_active":"2024-09-16T11:07:01.233205+0000","last_peered":"2024-09-16T11:07:01.233205+0000","last_clean":"2024-09-16T11:07:01.233205+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:07:01.233205+0000","last_undegraded":"2024-09-16T11:07:01.233205+0000","last_fullsized":"2024-09-16T11:07:01.233205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:39.706733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":633,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958113+0000","last_change":"2024-09-16T10:51:10.849321+0000","last_active":"2024-09-16T11:07:01.958113+0000","last_peered":"2024-09-16T11:07:01.958113+0000","last_clean":"2024-09-16T11:07:01.958113+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:07:01.958113+0000","last_undegraded":"2024-09-16T11:07:01.958113+0000","last_fullsized":"2024-09-16T11:07:01.958113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:42.862278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"20'7","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959504+0000","last_change":"2024-09-16T10:51:10.808306+0000","last_active":"2024-09-16T11:07:01.959504+0000","last_peered":"2024-09-16T11:07:01.959504+0000","last_clean":"2024-09-16T11:07:01.959504+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:07:01.959504+0000","last_undegraded":"2024-09-16T11:07:01.959504+0000","last_fullsized":"2024-09-16T11:07:01.959504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:18.223681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":664,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.960466+0000","last_change":"2024-09-16T10:51:10.869585+0000","last_active":"2024-09-16T11:07:01.960466+0000","last_peered":"2024-09-16T11:07:01.960466+0000","last_clean":"2024-09-16T11:07:01.960466+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:07:01.960466+0000","last_undegraded":"2024-09-16T11:07:01.960466+0000","last_fullsized":"2024-09-16T11:07:01.960466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:37.869026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232719+0000","last_change":"2024-09-16T10:51:10.814816+0000","last_active":"2024-09-16T11:07:01.232719+0000","last_peered":"2024-09-16T11:07:01.232719+0000","last_clean":"2024-09-16T11:07:01.232719+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:07:01.232719+0000","last_undegraded":"2024-09-16T11:07:01.232719+0000","last_fullsized":"2024-09-16T11:07:01.232719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:29:46.969757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80048,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291792,"statfs":{"total":96636764160,"available":96554795008,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.60299999999999998,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":1.123,"15min":1.984},"last":0.73099999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.80800000000000005},"min":{"1min":0.65800000000000003,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.3},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.74399999999999999,"5min":0.56999999999999995,"15min":0.53000000000000003},"max":{"1min":1.1459999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.75900000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.68100000000000005,"5min":0.54200000000000004,"15min":0.45400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.1779999999999999},"last":0.624},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":1.0620000000000001,"15min":1.22},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67600000000000005,"15min":0.73199999999999998},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.385},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82699999999999996,"15min":0.78900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.365},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.72299999999999998,"15min":0.71199999999999997},"min":{"1min":0.58999999999999997,"5min":0.53800000000000003,"15min":0.39400000000000002},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95399999999999996,"15min":0.92900000000000005},"min":{"1min":0.83399999999999996,"5min":0.72799999999999998,"15min":0.499},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.91800000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80040,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291800,"statfs":{"total":96636764160,"available":96554803200,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.78600000000000003,"15min":0.82899999999999996},"min":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.36599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":5.351},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80000000000000004,"15min":0.78000000000000003},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.29999999999999999},"max":{"1min":1.032,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90800000000000003,"15min":0.91800000000000004},"min":{"1min":0.623,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.268,"5min":1.268,"15min":10.134},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.91900000000000004,"15min":0.91200000000000003},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":0.89300000000000002},"min":{"1min":0.625,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.46899999999999997},"max":{"1min":1.044,"5min":1.28,"15min":1.3859999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.34100000000000003},"max":{"1min":0.86199999999999999,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":0.996,"5min":1.268,"15min":5.0579999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75900000000000001,"15min":0.77000000000000002},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.996,"5min":1.153,"15min":3.7770000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.83499999999999996},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":1.111,"15min":8.5559999999999992},"last":0.68300000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76476,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295364,"statfs":{"total":96636764160,"available":96558452736,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.37},"max":{"1min":0.97799999999999998,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.35599999999999998},"max":{"1min":0.93400000000000005,"5min":1.1000000000000001,"15min":1.609},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.871,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.436},"max":{"1min":1.01,"5min":1.4430000000000001,"15min":1.821},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.85499999999999998,"5min":0.64500000000000002,"15min":0.46899999999999997},"max":{"1min":1.101,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.67700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98899999999999999,"5min":1.2290000000000001,"15min":10.129},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.65400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.006,"5min":2.3559999999999999,"15min":10.449999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":3.4710000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.47999999999999998,"15min":0.33500000000000002},"max":{"1min":0.98399999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.749,"15min":0.77000000000000002},"min":{"1min":0.56200000000000006,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":1.179,"5min":1.1870000000000001,"15min":1.353},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81200000000000006,"15min":0.84199999999999997},"min":{"1min":0.76000000000000001,"5min":0.52200000000000002,"15min":0.39600000000000002},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":2.625},"last":0.75900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71848,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299992,"statfs":{"total":96636764160,"available":96563191808,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.39500000000000002},"max":{"1min":0.94899999999999995,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.65300000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.76900000000000002,"15min":0.746},"min":{"1min":0.63200000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":1.155,"15min":1.155},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0569999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.89400000000000002},"min":{"1min":0.71599999999999997,"5min":0.66100000000000003,"15min":0.60099999999999998},"max":{"1min":1.1739999999999999,"5min":1.341,"15min":1.774},"last":0.97499999999999998}]},{"osd":4,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88700000000000001,"15min":0.89900000000000002},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.59499999999999997},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90100000000000002,"15min":0.875},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59899999999999998},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":1.7090000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.88300000000000001,"15min":0.88100000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.141,"5min":1.397,"15min":3.7989999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.95499999999999996,"15min":0.95099999999999996},"min":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.57799999999999996},"max":{"1min":1.1950000000000001,"5min":1.204,"15min":3.2799999999999998},"last":0.84099999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84144,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":79486,"kb_avail":94287696,"statfs":{"total":96636764160,"available":96550600704,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81394119},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.72699999999999998},"min":{"1min":0.59499999999999997,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":1.04,"15min":1.845},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.51900000000000002,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":1.034,"5min":1.034,"15min":4.9260000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.89700000000000002,"15min":2.8370000000000002},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.879,"15min":0.89200000000000002},"min":{"1min":0.76400000000000001,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.95799999999999996,"15min":0.96799999999999997},"min":{"1min":0.746,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.208,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.89700000000000002},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.93899999999999995,"5min":1.2729999999999999,"15min":2.802},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94899999999999995,"15min":0.94199999999999995},"min":{"1min":0.76700000000000002,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.206,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.90800000000000003},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.71299999999999997,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.159,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.69399999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80564,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291276,"statfs":{"total":96636764160,"available":96554266624,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.221},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":2.6469999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.247},"max":{"1min":1.079,"5min":1.079,"15min":2.3479999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71499999999999997,"5min":1.0289999999999999,"15min":1.327},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":1.228,"15min":4.1319999999999997},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85799999999999998,"15min":0.90300000000000002},"min":{"1min":0.69399999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.70699999999999996,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":1.175,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.88700000000000001}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.88500000000000001,"15min":0.91700000000000004},"min":{"1min":0.73099999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.008,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.74299999999999999,"5min":0.46999999999999997,"15min":0.36699999999999999},"max":{"1min":1.1020000000000001,"5min":1.302,"15min":3.1440000000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.89200000000000002},"min":{"1min":0.69399999999999995,"5min":0.49399999999999999,"15min":0.45200000000000001},"max":{"1min":1.1479999999999999,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88,"15min":0.94499999999999995},"min":{"1min":0.69399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":1.0409999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:10.245 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T11:08:10.245 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: get_monmap_and_config 2024-09-16T11:08:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: monmap: 2024-09-16T11:08:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.356 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:10.354738+0000 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:10.354738+0000 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:10.357 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 Processor -- start 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 -- start start 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: init 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3fa409aa50 0x7f3fa409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-c con 0x7f3fa409aa50 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 0x7f3fa40a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-b con 0x7f3fa4071870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fa409ee20 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa409caa0 con 0x7f3fa409ee20 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa409e3a0 con 0x7f3fa409aa50 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa409ae30 con 0x7f3fa4071870 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.349+0000 7f3fa90d8700 10 monclient(hunting): authenticate will time out at 1948.931877s 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 0x7f3fa40a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa88d7700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3fa409ee20 0x7f3fa41397f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51304/0 (socket says 172.21.15.5:51304) 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa88d7700 1 -- 172.21.15.5:0/3818759000 learned_addr learned my addr 172.21.15.5:0/3818759000 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): get_auth_request con 0x7f3fa4071870 auth_method 0 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 1 --2- 172.21.15.5:0/3818759000 >> v2:172.21.15.143:3300/0 conn(0x7f3fa409aa50 0x7f3fa409e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): get_auth_request con 0x7f3fa409aa50 auth_method 0 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_done global_id 114137 payload 274 2024-09-16T11:08:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 -- 172.21.15.5:0/3818759000 >> v2:172.21.15.143:3300/0 conn(0x7f3fa409aa50 msgr2=0x7f3fa409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 --2- 172.21.15.5:0/3818759000 >> v2:172.21.15.143:3300/0 conn(0x7f3fa409aa50 0x7f3fa409e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 -- 172.21.15.5:0/3818759000 >> v1:172.21.15.5:6789/0 conn(0x7f3fa409ee20 legacy=0x7f3fa41397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 monclient: found mon.noname-b 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 1 -- 172.21.15.5:0/3818759000 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373788512 0 0) 0x7f3f9c001110 con 0x7f3fa409ee20 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 -- 172.21.15.5:0/3818759000 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa409b5f0 con 0x7f3fa4071870 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:08:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa37fe700 1 --2- 172.21.15.5:0/3818759000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 0x7f3fa40a10f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009850 tx=0x7f3f98009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4317a53e7e660afc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 1 -- 172.21.15.5:0/3818759000 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98011830 con 0x7f3fa4071870 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: dump: 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 1 -- 172.21.15.5:0/3818759000 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f98011990 con 0x7f3fa4071870 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 5 monclient: authenticate success, global_id 114137 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 1 -- 172.21.15.5:0/3818759000 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98011d40 con 0x7f3fa4071870 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 4 set_mon_vals no callback set 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa2ffd700 10 monclient: dump: 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 10 monclient: shutdown 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3818759000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 msgr2=0x7f3fa40a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3818759000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 0x7f3fa40a10f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009850 tx=0x7f3f98009b60 comp rx=0 tx=0).stop 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3818759000 shutdown_connections 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3818759000 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa4071870 0x7f3fa40a10f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3818759000 >> v2:172.21.15.143:3300/0 conn(0x7f3fa409aa50 0x7f3fa409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3818759000 >> 172.21.15.5:0/3818759000 conn(0x7f3fa40a45d0 msgr2=0x7f3fa40a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3818759000 shutdown_connections 2024-09-16T11:08:10.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3818759000 wait complete. 2024-09-16T11:08:10.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 10 monclient: monmap: 2024-09-16T11:08:10.373 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.373 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 librados: starting msgr at 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 librados: starting objecter 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.353+0000 7f3fa90d8700 1 Processor -- start 2024-09-16T11:08:10.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- start start 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 librados: setting wanted keys 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 librados: calling monclient init 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: init 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-a con 0x7f3fa4071870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa409aa50 0x7f3fa41c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-c con 0x7f3fa409aa50 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 0x7f3fa411ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): picked mon.noname-b con 0x7f3fa409ee20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4122380 con 0x7f3fa4071870 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa409b0d0 con 0x7f3fa409ee20 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4122bf0 con 0x7f3fa409aa50 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient(hunting): authenticate will time out at 1948.936891s 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa409aa50 0x7f3fa41c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59914/0 (socket says 172.21.15.5:59914) 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 -- 172.21.15.5:0/3345267570 learned_addr learned my addr 172.21.15.5:0/3345267570 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): get_auth_request con 0x7f3fa4071870 auth_method 0 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): get_auth_request con 0x7f3fa409aa50 auth_method 0 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 0x7f3fa411ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 10 monclient(hunting): get_auth_request con 0x7f3fa409ee20 auth_method 0 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient(hunting): handle_auth_done global_id 74688 payload 824 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 -- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa409aa50 msgr2=0x7f3fa41c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa409aa50 0x7f3fa41c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 -- 172.21.15.5:0/3345267570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 msgr2=0x7f3fa411ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 0x7f3fa411ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 monclient: found mon.noname-a 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 -- 172.21.15.5:0/3345267570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f980094b0 con 0x7f3fa4071870 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa88d7700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 0x7f3fa411ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa3fff700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400efc0 tx=0x7f3f9400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 client.?.objecter ms_handle_connect 0x7f3fa4071870 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400cdc0 con 0x7f3fa4071870 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: dump: 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.379 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 5 monclient: authenticate success, global_id 74688 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9400f040 con 0x7f3fa4071870 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f940146a0 con 0x7f3fa4071870 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _renew_subs 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa41211e0 con 0x7f3fa4071870 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:10.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa0ff9700 10 monclient: dump: 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa27fc700 4 set_mon_vals no callback set 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 client.74688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _renew_subs 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa40a3760 con 0x7f3fa4071870 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 librados: init done 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.357+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa409e8e0 con 0x7f3fa4071870 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3f94014800 con 0x7f3fa4071870 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 1 --2- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 0x7f3f90039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f3f940487e0 con 0x7f3fa4071870 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 10 client.74688.objecter ms_dispatch 0x7f3fa4000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 3 client.74688.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 3 client.74688.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 20 client.74688.objecter dump_active .. 0 homeless 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa37fe700 1 --2- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 0x7f3f90039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa37fe700 10 monclient: get_auth_request con 0x7f3f90036f60 auth_method 0 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa37fe700 1 --2- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 0x7f3f90039420 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7f3f980090b0 tx=0x7f3f98008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.361+0000 7f3fa0ff9700 10 client.74688.objecter ms_handle_connect 0x7f3f90036f60 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.365+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3f94007d90 con 0x7f3fa4071870 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.365+0000 7f3fa0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.365+0000 7f3fa0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:10.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.529+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fa409d580 con 0x7f3f90036f60 2024-09-16T11:08:10.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.537+0000 7f3fa0ff9700 1 -- 172.21.15.5:0/3345267570 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133861 (secure 0 0 0) 0x7f3f98011c00 con 0x7f3f90036f60 2024-09-16T11:08:10.549 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 10 librados: watch_flush enter 2024-09-16T11:08:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 10 librados: watch_flush exit 2024-09-16T11:08:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 20 client.74688.objecter shutdown clearing up homeless session... 2024-09-16T11:08:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 10 client.74688.objecter successfully canceled tick 2024-09-16T11:08:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 msgr2=0x7f3f90039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 0x7f3f90039420 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7f3f980090b0 tx=0x7f3f98008030 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 10 monclient: shutdown 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 msgr2=0x7f3fa41c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3f9400efc0 tx=0x7f3f9400c3a0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 shutdown_connections 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa409ee20 0x7f3fa411ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3fa409aa50 0x7f3fa41c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3f90036f60 0x7f3f90039420 unknown :-1 s=CLOSED pgs=3913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 --2- 172.21.15.5:0/3345267570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3fa4071870 0x7f3fa41c8180 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 >> 172.21.15.5:0/3345267570 conn(0x7f3fa40a45d0 msgr2=0x7f3fa4070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 shutdown_connections 2024-09-16T11:08:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 -- 172.21.15.5:0/3345267570 wait complete. 2024-09-16T11:08:10.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.541+0000 7f3fa90d8700 1 librados: shutdown 2024-09-16T11:08:10.553 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:10.565 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1234,"stamp":"2024-09-16T11:08:09.666457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":473120,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":444150,"kb_avail":565757920,"statfs":{"total":579820584960,"available":579336110080,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":454810278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003945"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":663,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104625+0000","last_change":"2024-09-16T10:51:10.895648+0000","last_active":"2024-09-16T11:07:01.104625+0000","last_peered":"2024-09-16T11:07:01.104625+0000","last_clean":"2024-09-16T11:07:01.104625+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:07:01.104625+0000","last_undegraded":"2024-09-16T11:07:01.104625+0000","last_fullsized":"2024-09-16T11:07:01.104625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:14.539597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":636,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221962+0000","last_change":"2024-09-16T10:51:10.811863+0000","last_active":"2024-09-16T11:07:00.221962+0000","last_peered":"2024-09-16T11:07:00.221962+0000","last_clean":"2024-09-16T11:07:00.221962+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:07:00.221962+0000","last_undegraded":"2024-09-16T11:07:00.221962+0000","last_fullsized":"2024-09-16T11:07:00.221962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:51.833056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053499+0000","last_change":"2024-09-16T10:51:10.836991+0000","last_active":"2024-09-16T11:07:00.053499+0000","last_peered":"2024-09-16T11:07:00.053499+0000","last_clean":"2024-09-16T11:07:00.053499+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:07:00.053499+0000","last_undegraded":"2024-09-16T11:07:00.053499+0000","last_fullsized":"2024-09-16T11:07:00.053499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:54.839408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.870842+0000","last_change":"2024-09-16T10:51:10.861754+0000","last_active":"2024-09-16T11:07:01.870842+0000","last_peered":"2024-09-16T11:07:01.870842+0000","last_clean":"2024-09-16T11:07:01.870842+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:07:01.870842+0000","last_undegraded":"2024-09-16T11:07:01.870842+0000","last_fullsized":"2024-09-16T11:07:01.870842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:10.098818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054257+0000","last_change":"2024-09-16T10:51:10.888616+0000","last_active":"2024-09-16T11:07:00.054257+0000","last_peered":"2024-09-16T11:07:00.054257+0000","last_clean":"2024-09-16T11:07:00.054257+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:07:00.054257+0000","last_undegraded":"2024-09-16T11:07:00.054257+0000","last_fullsized":"2024-09-16T11:07:00.054257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:47.053233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551276+0000","last_change":"2024-09-16T10:51:10.820735+0000","last_active":"2024-09-16T11:07:00.551276+0000","last_peered":"2024-09-16T11:07:00.551276+0000","last_clean":"2024-09-16T11:07:00.551276+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:07:00.551276+0000","last_undegraded":"2024-09-16T11:07:00.551276+0000","last_fullsized":"2024-09-16T11:07:00.551276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:00.585239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053844+0000","last_change":"2024-09-16T10:51:10.906082+0000","last_active":"2024-09-16T11:07:00.053844+0000","last_peered":"2024-09-16T11:07:00.053844+0000","last_clean":"2024-09-16T11:07:00.053844+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:07:00.053844+0000","last_undegraded":"2024-09-16T11:07:00.053844+0000","last_fullsized":"2024-09-16T11:07:00.053844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:24.987446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051232+0000","last_change":"2024-09-16T10:51:10.825984+0000","last_active":"2024-09-16T11:07:00.051232+0000","last_peered":"2024-09-16T11:07:00.051232+0000","last_clean":"2024-09-16T11:07:00.051232+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:07:00.051232+0000","last_undegraded":"2024-09-16T11:07:00.051232+0000","last_fullsized":"2024-09-16T11:07:00.051232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:31:17.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551587+0000","last_change":"2024-09-16T10:51:10.851590+0000","last_active":"2024-09-16T11:07:00.551587+0000","last_peered":"2024-09-16T11:07:00.551587+0000","last_clean":"2024-09-16T11:07:00.551587+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:07:00.551587+0000","last_undegraded":"2024-09-16T11:07:00.551587+0000","last_fullsized":"2024-09-16T11:07:00.551587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:52.807901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":637,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221819+0000","last_change":"2024-09-16T10:51:10.818746+0000","last_active":"2024-09-16T11:07:00.221819+0000","last_peered":"2024-09-16T11:07:00.221819+0000","last_clean":"2024-09-16T11:07:00.221819+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:07:00.221819+0000","last_undegraded":"2024-09-16T11:07:00.221819+0000","last_fullsized":"2024-09-16T11:07:00.221819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.511253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961018+0000","last_change":"2024-09-16T10:51:10.916719+0000","last_active":"2024-09-16T11:07:01.961018+0000","last_peered":"2024-09-16T11:07:01.961018+0000","last_clean":"2024-09-16T11:07:01.961018+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:07:01.961018+0000","last_undegraded":"2024-09-16T11:07:01.961018+0000","last_fullsized":"2024-09-16T11:07:01.961018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:07.031545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":647,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959927+0000","last_change":"2024-09-16T10:51:10.807192+0000","last_active":"2024-09-16T11:07:01.959927+0000","last_peered":"2024-09-16T11:07:01.959927+0000","last_clean":"2024-09-16T11:07:01.959927+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:07:01.959927+0000","last_undegraded":"2024-09-16T11:07:01.959927+0000","last_fullsized":"2024-09-16T11:07:01.959927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:17.091636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958919+0000","last_change":"2024-09-16T10:51:10.888609+0000","last_active":"2024-09-16T11:07:01.958919+0000","last_peered":"2024-09-16T11:07:01.958919+0000","last_clean":"2024-09-16T11:07:01.958919+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:07:01.958919+0000","last_undegraded":"2024-09-16T11:07:01.958919+0000","last_fullsized":"2024-09-16T11:07:01.958919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:51:59.284621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233208+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T11:07:01.233208+0000","last_peered":"2024-09-16T11:07:01.233208+0000","last_clean":"2024-09-16T11:07:01.233208+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:07:01.233208+0000","last_undegraded":"2024-09-16T11:07:01.233208+0000","last_fullsized":"2024-09-16T11:07:01.233208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961200+0000","last_change":"2024-09-16T10:51:10.907339+0000","last_active":"2024-09-16T11:07:01.961200+0000","last_peered":"2024-09-16T11:07:01.961200+0000","last_clean":"2024-09-16T11:07:01.961200+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:07:01.961200+0000","last_undegraded":"2024-09-16T11:07:01.961200+0000","last_fullsized":"2024-09-16T11:07:01.961200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:22.367946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959643+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T11:07:01.959643+0000","last_peered":"2024-09-16T11:07:01.959643+0000","last_clean":"2024-09-16T11:07:01.959643+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:07:01.959643+0000","last_undegraded":"2024-09-16T11:07:01.959643+0000","last_fullsized":"2024-09-16T11:07:01.959643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":554,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051532+0000","last_change":"2024-09-16T10:52:56.726039+0000","last_active":"2024-09-16T11:07:00.051532+0000","last_peered":"2024-09-16T11:07:00.051532+0000","last_clean":"2024-09-16T11:07:00.051532+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:07:00.051532+0000","last_undegraded":"2024-09-16T11:07:00.051532+0000","last_fullsized":"2024-09-16T11:07:00.051532+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.112159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":648,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221678+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T11:07:00.221678+0000","last_peered":"2024-09-16T11:07:00.221678+0000","last_clean":"2024-09-16T11:07:00.221678+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:07:00.221678+0000","last_undegraded":"2024-09-16T11:07:00.221678+0000","last_fullsized":"2024-09-16T11:07:00.221678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961095+0000","last_change":"2024-09-16T10:51:10.915110+0000","last_active":"2024-09-16T11:07:01.961095+0000","last_peered":"2024-09-16T11:07:01.961095+0000","last_clean":"2024-09-16T11:07:01.961095+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:07:01.961095+0000","last_undegraded":"2024-09-16T11:07:01.961095+0000","last_fullsized":"2024-09-16T11:07:01.961095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:54.203725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":658,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051747+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T11:07:00.051747+0000","last_peered":"2024-09-16T11:07:00.051747+0000","last_clean":"2024-09-16T11:07:00.051747+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:07:00.051747+0000","last_undegraded":"2024-09-16T11:07:00.051747+0000","last_fullsized":"2024-09-16T11:07:00.051747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":677,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104372+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T11:07:01.104372+0000","last_peered":"2024-09-16T11:07:01.104372+0000","last_clean":"2024-09-16T11:07:01.104372+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:07:01.104372+0000","last_undegraded":"2024-09-16T11:07:01.104372+0000","last_fullsized":"2024-09-16T11:07:01.104372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958388+0000","last_change":"2024-09-16T10:51:10.900283+0000","last_active":"2024-09-16T11:07:01.958388+0000","last_peered":"2024-09-16T11:07:01.958388+0000","last_clean":"2024-09-16T11:07:01.958388+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:07:01.958388+0000","last_undegraded":"2024-09-16T11:07:01.958388+0000","last_fullsized":"2024-09-16T11:07:01.958388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:51.095223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":644,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551103+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T11:07:00.551103+0000","last_peered":"2024-09-16T11:07:00.551103+0000","last_clean":"2024-09-16T11:07:00.551103+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:07:00.551103+0000","last_undegraded":"2024-09-16T11:07:00.551103+0000","last_fullsized":"2024-09-16T11:07:00.551103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959213+0000","last_change":"2024-09-16T10:51:10.777356+0000","last_active":"2024-09-16T11:07:01.959213+0000","last_peered":"2024-09-16T11:07:01.959213+0000","last_clean":"2024-09-16T11:07:01.959213+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:07:01.959213+0000","last_undegraded":"2024-09-16T11:07:01.959213+0000","last_fullsized":"2024-09-16T11:07:01.959213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:54.038836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232503+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T11:07:01.232503+0000","last_peered":"2024-09-16T11:07:01.232503+0000","last_clean":"2024-09-16T11:07:01.232503+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:07:01.232503+0000","last_undegraded":"2024-09-16T11:07:01.232503+0000","last_fullsized":"2024-09-16T11:07:01.232503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232530+0000","last_change":"2024-09-16T10:51:10.824359+0000","last_active":"2024-09-16T11:07:01.232530+0000","last_peered":"2024-09-16T11:07:01.232530+0000","last_clean":"2024-09-16T11:07:01.232530+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:07:01.232530+0000","last_undegraded":"2024-09-16T11:07:01.232530+0000","last_fullsized":"2024-09-16T11:07:01.232530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:09.821124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":676,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104414+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T11:07:01.104414+0000","last_peered":"2024-09-16T11:07:01.104414+0000","last_clean":"2024-09-16T11:07:01.104414+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:07:01.104414+0000","last_undegraded":"2024-09-16T11:07:01.104414+0000","last_fullsized":"2024-09-16T11:07:01.104414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":797,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232630+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T11:07:01.232630+0000","last_peered":"2024-09-16T11:07:01.232630+0000","last_clean":"2024-09-16T11:07:01.232630+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:07:01.232630+0000","last_undegraded":"2024-09-16T11:07:01.232630+0000","last_fullsized":"2024-09-16T11:07:01.232630+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232619+0000","last_change":"2024-09-16T10:51:10.819816+0000","last_active":"2024-09-16T11:07:01.232619+0000","last_peered":"2024-09-16T11:07:01.232619+0000","last_clean":"2024-09-16T11:07:01.232619+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:07:01.232619+0000","last_undegraded":"2024-09-16T11:07:01.232619+0000","last_fullsized":"2024-09-16T11:07:01.232619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:01.570513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232841+0000","last_change":"2024-09-16T10:51:10.881698+0000","last_active":"2024-09-16T11:07:01.232841+0000","last_peered":"2024-09-16T11:07:01.232841+0000","last_clean":"2024-09-16T11:07:01.232841+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:07:01.232841+0000","last_undegraded":"2024-09-16T11:07:01.232841+0000","last_fullsized":"2024-09-16T11:07:01.232841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:17.134723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959641+0000","last_change":"2024-09-16T10:51:10.823091+0000","last_active":"2024-09-16T11:07:01.959641+0000","last_peered":"2024-09-16T11:07:01.959641+0000","last_clean":"2024-09-16T11:07:01.959641+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:07:01.959641+0000","last_undegraded":"2024-09-16T11:07:01.959641+0000","last_fullsized":"2024-09-16T11:07:01.959641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:32.535561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551175+0000","last_change":"2024-09-16T10:51:10.890050+0000","last_active":"2024-09-16T11:07:00.551175+0000","last_peered":"2024-09-16T11:07:00.551175+0000","last_clean":"2024-09-16T11:07:00.551175+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:07:00.551175+0000","last_undegraded":"2024-09-16T11:07:00.551175+0000","last_fullsized":"2024-09-16T11:07:00.551175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:00.950193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":649,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054067+0000","last_change":"2024-09-16T10:51:10.851438+0000","last_active":"2024-09-16T11:07:00.054067+0000","last_peered":"2024-09-16T11:07:00.054067+0000","last_clean":"2024-09-16T11:07:00.054067+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:07:00.054067+0000","last_undegraded":"2024-09-16T11:07:00.054067+0000","last_fullsized":"2024-09-16T11:07:00.054067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:26.462001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":539,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871448+0000","last_change":"2024-09-16T10:52:56.729849+0000","last_active":"2024-09-16T11:07:01.871448+0000","last_peered":"2024-09-16T11:07:01.871448+0000","last_clean":"2024-09-16T11:07:01.871448+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:07:01.871448+0000","last_undegraded":"2024-09-16T11:07:01.871448+0000","last_fullsized":"2024-09-16T11:07:01.871448+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:09.227854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":635,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221534+0000","last_change":"2024-09-16T10:51:10.773626+0000","last_active":"2024-09-16T11:07:00.221534+0000","last_peered":"2024-09-16T11:07:00.221534+0000","last_clean":"2024-09-16T11:07:00.221534+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:07:00.221534+0000","last_undegraded":"2024-09-16T11:07:00.221534+0000","last_fullsized":"2024-09-16T11:07:00.221534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:31.797539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":631,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871689+0000","last_change":"2024-09-16T10:51:10.895063+0000","last_active":"2024-09-16T11:07:01.871689+0000","last_peered":"2024-09-16T11:07:01.871689+0000","last_clean":"2024-09-16T11:07:01.871689+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:07:01.871689+0000","last_undegraded":"2024-09-16T11:07:01.871689+0000","last_fullsized":"2024-09-16T11:07:01.871689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:59.114090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":636,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233205+0000","last_change":"2024-09-16T10:51:10.830383+0000","last_active":"2024-09-16T11:07:01.233205+0000","last_peered":"2024-09-16T11:07:01.233205+0000","last_clean":"2024-09-16T11:07:01.233205+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:07:01.233205+0000","last_undegraded":"2024-09-16T11:07:01.233205+0000","last_fullsized":"2024-09-16T11:07:01.233205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:39.706733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":633,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958113+0000","last_change":"2024-09-16T10:51:10.849321+0000","last_active":"2024-09-16T11:07:01.958113+0000","last_peered":"2024-09-16T11:07:01.958113+0000","last_clean":"2024-09-16T11:07:01.958113+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:07:01.958113+0000","last_undegraded":"2024-09-16T11:07:01.958113+0000","last_fullsized":"2024-09-16T11:07:01.958113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:42.862278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"20'7","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959504+0000","last_change":"2024-09-16T10:51:10.808306+0000","last_active":"2024-09-16T11:07:01.959504+0000","last_peered":"2024-09-16T11:07:01.959504+0000","last_clean":"2024-09-16T11:07:01.959504+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:07:01.959504+0000","last_undegraded":"2024-09-16T11:07:01.959504+0000","last_fullsized":"2024-09-16T11:07:01.959504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:18.223681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":664,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.960466+0000","last_change":"2024-09-16T10:51:10.869585+0000","last_active":"2024-09-16T11:07:01.960466+0000","last_peered":"2024-09-16T11:07:01.960466+0000","last_clean":"2024-09-16T11:07:01.960466+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:07:01.960466+0000","last_undegraded":"2024-09-16T11:07:01.960466+0000","last_fullsized":"2024-09-16T11:07:01.960466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:37.869026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232719+0000","last_change":"2024-09-16T10:51:10.814816+0000","last_active":"2024-09-16T11:07:01.232719+0000","last_peered":"2024-09-16T11:07:01.232719+0000","last_clean":"2024-09-16T11:07:01.232719+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:07:01.232719+0000","last_undegraded":"2024-09-16T11:07:01.232719+0000","last_fullsized":"2024-09-16T11:07:01.232719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:29:46.969757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80048,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291792,"statfs":{"total":96636764160,"available":96554795008,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.60299999999999998,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":1.123,"15min":1.984},"last":0.73099999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.80800000000000005},"min":{"1min":0.65800000000000003,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.3},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.74399999999999999,"5min":0.56999999999999995,"15min":0.53000000000000003},"max":{"1min":1.1459999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.75900000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.68100000000000005,"5min":0.54200000000000004,"15min":0.45400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.1779999999999999},"last":0.624},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":1.0620000000000001,"15min":1.22},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67600000000000005,"15min":0.73199999999999998},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.385},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82699999999999996,"15min":0.78900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.365},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.72299999999999998,"15min":0.71199999999999997},"min":{"1min":0.58999999999999997,"5min":0.53800000000000003,"15min":0.39400000000000002},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95399999999999996,"15min":0.92900000000000005},"min":{"1min":0.83399999999999996,"5min":0.72799999999999998,"15min":0.499},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.91800000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80040,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291800,"statfs":{"total":96636764160,"available":96554803200,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.78600000000000003,"15min":0.82899999999999996},"min":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.36599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":5.351},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80000000000000004,"15min":0.78000000000000003},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.29999999999999999},"max":{"1min":1.032,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90800000000000003,"15min":0.91800000000000004},"min":{"1min":0.623,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.268,"5min":1.268,"15min":10.134},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.91900000000000004,"15min":0.91200000000000003},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":0.89300000000000002},"min":{"1min":0.625,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.46899999999999997},"max":{"1min":1.044,"5min":1.28,"15min":1.3859999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.34100000000000003},"max":{"1min":0.86199999999999999,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":0.996,"5min":1.268,"15min":5.0579999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75900000000000001,"15min":0.77000000000000002},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.996,"5min":1.153,"15min":3.7770000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.83499999999999996},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":1.111,"15min":8.5559999999999992},"last":0.68300000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76476,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295364,"statfs":{"total":96636764160,"available":96558452736,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.37},"max":{"1min":0.97799999999999998,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.35599999999999998},"max":{"1min":0.93400000000000005,"5min":1.1000000000000001,"15min":1.609},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.871,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.436},"max":{"1min":1.01,"5min":1.4430000000000001,"15min":1.821},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.85499999999999998,"5min":0.64500000000000002,"15min":0.46899999999999997},"max":{"1min":1.101,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.67700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98899999999999999,"5min":1.2290000000000001,"15min":10.129},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.65400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.006,"5min":2.3559999999999999,"15min":10.449999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":3.4710000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.47999999999999998,"15min":0.33500000000000002},"max":{"1min":0.98399999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.749,"15min":0.77000000000000002},"min":{"1min":0.56200000000000006,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":1.179,"5min":1.1870000000000001,"15min":1.353},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81200000000000006,"15min":0.84199999999999997},"min":{"1min":0.76000000000000001,"5min":0.52200000000000002,"15min":0.39600000000000002},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":2.625},"last":0.75900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71848,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299992,"statfs":{"total":96636764160,"available":96563191808,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.39500000000000002},"max":{"1min":0.94899999999999995,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.65300000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.76900000000000002,"15min":0.746},"min":{"1min":0.63200000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":1.155,"15min":1.155},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0569999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.89400000000000002},"min":{"1min":0.71599999999999997,"5min":0.66100000000000003,"15min":0.60099999999999998},"max":{"1min":1.1739999999999999,"5min":1.341,"15min":1.774},"last":0.97499999999999998}]},{"osd":4,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88700000000000001,"15min":0.89900000000000002},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.59499999999999997},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90100000000000002,"15min":0.875},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59899999999999998},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":1.7090000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.88300000000000001,"15min":0.88100000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.141,"5min":1.397,"15min":3.7989999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.95499999999999996,"15min":0.95099999999999996},"min":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.57799999999999996},"max":{"1min":1.1950000000000001,"5min":1.204,"15min":3.2799999999999998},"last":0.84099999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84144,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":79486,"kb_avail":94287696,"statfs":{"total":96636764160,"available":96550600704,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81394119},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.72699999999999998},"min":{"1min":0.59499999999999997,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":1.04,"15min":1.845},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.51900000000000002,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":1.034,"5min":1.034,"15min":4.9260000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.89700000000000002,"15min":2.8370000000000002},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.879,"15min":0.89200000000000002},"min":{"1min":0.76400000000000001,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.95799999999999996,"15min":0.96799999999999997},"min":{"1min":0.746,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.208,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.89700000000000002},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.93899999999999995,"5min":1.2729999999999999,"15min":2.802},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94899999999999995,"15min":0.94199999999999995},"min":{"1min":0.76700000000000002,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.206,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.90800000000000003},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.71299999999999997,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.159,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.69399999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80564,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291276,"statfs":{"total":96636764160,"available":96554266624,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.221},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":2.6469999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.247},"max":{"1min":1.079,"5min":1.079,"15min":2.3479999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71499999999999997,"5min":1.0289999999999999,"15min":1.327},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":1.228,"15min":4.1319999999999997},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85799999999999998,"15min":0.90300000000000002},"min":{"1min":0.69399999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.70699999999999996,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":1.175,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.88700000000000001}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.88500000000000001,"15min":0.91700000000000004},"min":{"1min":0.73099999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.008,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.74299999999999999,"5min":0.46999999999999997,"15min":0.36699999999999999},"max":{"1min":1.1020000000000001,"5min":1.302,"15min":3.1440000000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.89200000000000002},"min":{"1min":0.69399999999999995,"5min":0.49399999999999999,"15min":0.45200000000000001},"max":{"1min":1.1479999999999999,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88,"15min":0.94499999999999995},"min":{"1min":0.69399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":1.0409999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:10.569 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: get_monmap_and_config 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: monmap: 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:10.677501+0000 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:10.677501+0000 2024-09-16T11:08:10.679 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:10.680 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 Processor -- start 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 -- start start 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: init 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T11:08:10.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3eb009e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3eb009d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3eb009dfd0 0x7f3eb013c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-c con 0x7f3eb009dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb0070e40 con 0x7f3eb009d6a0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb009fef0 con 0x7f3eb009e8e0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb00a01d0 con 0x7f3eb009dfd0 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 10 monclient(hunting): authenticate will time out at 1949.253909s 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3eb009d6a0 0x7f3eb0138b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51316/0 (socket says 172.21.15.5:51316) 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 learned_addr learned my addr 172.21.15.5:0/3376185387 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:10.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/3376185387 >> v2:172.21.15.143:3300/0 conn(0x7f3eb009dfd0 0x7f3eb013c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request con 0x7f3eb009dfd0 auth_method 0 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721959041 0 0) 0x7f3eb0070e40 con 0x7f3eb009d6a0 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): _negotiate 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): my global_id is 74691 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c0035d0 con 0x7f3eb009d6a0 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948432100 0 0) 0x7f3e9c0035d0 con 0x7f3eb009d6a0 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 >> v2:172.21.15.143:3300/0 conn(0x7f3eb009dfd0 msgr2=0x7f3eb013c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> v2:172.21.15.143:3300/0 conn(0x7f3eb009dfd0 0x7f3eb013c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 >> v1:172.21.15.143:6789/0 conn(0x7f3eb009e8e0 legacy=0x7f3eb009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 monclient: found mon.noname-a-legacy 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb009f9f0 con 0x7f3eb009d6a0 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 5 monclient: authenticate success, global_id 74691 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f3e98001670 con 0x7f3eb009d6a0 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: dump: 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.689 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.690 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.690 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T11:08:10.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 >> v1:172.21.15.5:6789/0 conn(0x7f3eb009d6a0 legacy=0x7f3eb0138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:10.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003a20 0x7f3e9c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.b con 0x7f3e9c003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.a con 0x7f3e9c004870 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c005230 0x7f3e9c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.c con 0x7f3e9c005230 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00a01d0 con 0x7f3e9c004870 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00c180 con 0x7f3e9c003a20 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00c340 con 0x7f3e9c005230 2024-09-16T11:08:10.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb54a8700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c005230 0x7f3e9c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request con 0x7f3e9c005230 auth_method 0 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request con 0x7f3e9c004870 auth_method 0 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3ea77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.673+0000 7f3eaffff700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003a20 0x7f3e9c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): get_auth_request con 0x7f3e9c003a20 auth_method 0 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_done global_id 74691 payload 293 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c005230 msgr2=0x7f3e9c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c005230 0x7f3e9c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003a20 msgr2=0x7f3e9c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003a20 0x7f3e9c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 monclient: found mon.a 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c0035d0 con 0x7f3e9c004870 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c005230 0x7f3e9c00ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3eb4ca7700 1 -- 172.21.15.5:0/3376185387 reap_dead start 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 msgr2=0x7f3e9c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 msgr2=0x7f3e9c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f3e98005130 tx=0x7f3e9c0058a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T11:08:10.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.677+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient: get_auth_request con 0x7f3e9c004870 auth_method 0 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 0 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: ms_handle_reset current mon [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 msgr2=0x7f3e9c004cf0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c004870 0x7f3e9c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3e9c009a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.b con 0x7f3e9c003e00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3e9c00a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.a con 0x7f3e9c00a010 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3e9c00b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): picked mon.c con 0x7f3e9c00ac20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00b9b0 con 0x7f3e9c00a010 2024-09-16T11:08:10.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00bb20 con 0x7f3e9c003e00 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00bce0 con 0x7f3e9c00ac20 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3e9c00a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request con 0x7f3e9c00a010 auth_method 0 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3e9c00b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3e9c009a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): get_auth_request con 0x7f3e9c003e00 auth_method 0 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request con 0x7f3e9c00ac20 auth_method 0 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_done global_id 74691 payload 293 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 msgr2=0x7f3e9c00b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3e9c00b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 msgr2=0x7f3e9c009a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3e9c009a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 monclient: found mon.a 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:10.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 -- 172.21.15.5:0/3376185387 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f3eb0070e40 con 0x7f3e9c00a010 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3e9c00a3f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e98005130 tx=0x7f3e98008ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464fca6aa7a65101 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98005b80 con 0x7f3e9c00a010 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: dump: 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.894 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 1 -- 172.21.15.5:0/3376185387 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98005ce0 con 0x7f3e9c00a010 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb54a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eaf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb54a8700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:10.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb54a8700 4 set_mon_vals no callback set 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.881+0000 7f3eb54a8700 10 monclient: shutdown 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 msgr2=0x7f3e9c00a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3e9c00a3f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e98005130 tx=0x7f3e98008ca0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- 172.21.15.5:0/3376185387 shutdown_connections 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3e9c00b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3e9c00a3f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/3376185387 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3e9c009a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- 172.21.15.5:0/3376185387 >> 172.21.15.5:0/3376185387 conn(0x7f3eb00a45d0 msgr2=0x7f3eb00a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- 172.21.15.5:0/3376185387 shutdown_connections 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- 172.21.15.5:0/3376185387 wait complete. 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 10 monclient: build_initial_monmap 2024-09-16T11:08:10.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 10 monclient: monmap: 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 librados: starting msgr at 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 librados: starting objecter 2024-09-16T11:08:10.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 Processor -- start 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 -- start start 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 librados: setting wanted keys 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 1 librados: calling monclient init 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.885+0000 7f3eb54a8700 10 monclient: init 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3eb01223d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c003e00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c00a010 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3eb0122e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): picked mon.noname-c con 0x7f3e9c00ac20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0133c00 con 0x7f3e9c00a010 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb0126440 con 0x7f3e9c003e00 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb009bcf0 con 0x7f3e9c00ac20 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient(hunting): authenticate will time out at 1949.468412s 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3eb0122e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59970/0 (socket says 172.21.15.5:59970) 2024-09-16T11:08:10.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 1 -- 172.21.15.5:0/1814754020 learned_addr learned my addr 172.21.15.5:0/1814754020 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request con 0x7f3e9c00a010 auth_method 0 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request con 0x7f3e9c00ac20 auth_method 0 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3eb01223d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): get_auth_request con 0x7f3e9c003e00 auth_method 0 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient(hunting): handle_auth_done global_id 114143 payload 824 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 -- 172.21.15.5:0/1814754020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 msgr2=0x7f3eb01223d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3eb01223d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 -- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 msgr2=0x7f3eb0122910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient: _finish_hunting 0 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 monclient: found mon.noname-c 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 -- 172.21.15.5:0/1814754020 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb009b0d0 con 0x7f3e9c00ac20 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ea77fe700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb4ca7700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3eb0122e50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00a1150 tx=0x7f3ea8012580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 client.?.objecter ms_handle_connect 0x7f3e9c00ac20 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800e040 con 0x7f3e9c00ac20 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: dump: 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: _finish_auth 0 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: _check_auth_tickets 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800f040 con 0x7f3e9c00ac20 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 5 monclient: authenticate success, global_id 114143 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _renew_subs 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb0126020 con 0x7f3e9c00ac20 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8012c50 con 0x7f3e9c00ac20 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3ead7fa700 10 monclient: dump: 2024-09-16T11:08:10.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eaeffd700 4 set_mon_vals no callback set 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 client.114143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _renew_subs 2024-09-16T11:08:10.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb0123c50 con 0x7f3e9c00ac20 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.889+0000 7f3eb54a8700 1 librados: init done 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3ea8004030 con 0x7f3e9c00ac20 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eb54a8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eb54a8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eb54a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb009efc0 con 0x7f3e9c00ac20 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 1 --2- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 0x7f3e90039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f3ea801a070 con 0x7f3e9c00ac20 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 10 client.114143.objecter ms_dispatch 0x7f3eb0000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 3 client.114143.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 3 client.114143.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:10.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 20 client.114143.objecter dump_active .. 0 homeless 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eaffff700 1 --2- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 0x7f3e90039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eaffff700 10 monclient: get_auth_request con 0x7f3e90036f60 auth_method 0 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3eaffff700 1 --2- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 0x7f3e90039420 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0007fd0 tx=0x7f3ea0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.893+0000 7f3ead7fa700 10 client.114143.objecter ms_handle_connect 0x7f3e90036f60 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.901+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3ea8009c10 con 0x7f3e9c00ac20 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.901+0000 7f3ead7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:10.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:10.901+0000 7f3ead7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.073+0000 7f3eb54a8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.073+0000 7f3eb54a8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.073+0000 7f3eb54a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.073+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eb00721d0 con 0x7f3e9c00ac20 2024-09-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.077+0000 7f3ead7fa700 1 -- 172.21.15.5:0/1814754020 <== mon.2 v2:172.21.15.5:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v694) v1 ==== 74+0+13654 (secure 0 0 0) 0x7f3ea801d380 con 0x7f3e9c00ac20 2024-09-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.077+0000 7f3ead7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.077+0000 7f3ead7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:11.085 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":694,"fsid":"0ea84356-fa95-422b-b956-38c4a6b7c43e","created":"2024-09-16T10:49:49.828103+0000","modified":"2024-09-16T11:07:01.213883+0000","last_up_change":"2024-09-16T10:49:58.356658+0000","last_in_change":"2024-09-16T10:49:51.398636+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":90,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":260,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T10:49:59.343692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T10:49:59.729840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-16T10:51:08.838326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"208764a9-c9f7-49df-9201-b8a2b31be29b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2408116882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":2408116882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":2408116882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":2408116882}]},"public_addr":"172.21.15.5:6808/2408116882","cluster_addr":"172.21.15.5:6809/2408116882","heartbeat_back_addr":"172.21.15.5:6811/2408116882","heartbeat_front_addr":"172.21.15.5:6810/2408116882","state":["exists","up"]},{"osd":1,"uuid":"87c369a2-92fc-4740-b6d8-9ab060feea0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3675136489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":3675136489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":3675136489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":3675136489}]},"public_addr":"172.21.15.5:6800/3675136489","cluster_addr":"172.21.15.5:6801/3675136489","heartbeat_back_addr":"172.21.15.5:6803/3675136489","heartbeat_front_addr":"172.21.15.5:6802/3675136489","state":["exists","up"]},{"osd":2,"uuid":"909be8cc-8343-47b5-aa24-41ad442b8837","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2015722671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":2015722671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":2015722671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2015722671}]},"public_addr":"172.21.15.5:6804/2015722671","cluster_addr":"172.21.15.5:6805/2015722671","heartbeat_back_addr":"172.21.15.5:6807/2015722671","heartbeat_front_addr":"172.21.15.5:6806/2015722671","state":["exists","up"]},{"osd":3,"uuid":"bdaac9ae-7e26-471d-a9ed-95855c346e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3342511476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6801","nonce":3342511476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6803","nonce":3342511476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3342511476}]},"public_addr":"172.21.15.143:6800/3342511476","cluster_addr":"172.21.15.143:6801/3342511476","heartbeat_back_addr":"172.21.15.143:6803/3342511476","heartbeat_front_addr":"172.21.15.143:6802/3342511476","state":["exists","up"]},{"osd":4,"uuid":"4d9fc719-195a-41dc-bb5f-13b662146355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3348750750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":3348750750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6811","nonce":3348750750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3348750750}]},"public_addr":"172.21.15.143:6808/3348750750","cluster_addr":"172.21.15.143:6809/3348750750","heartbeat_back_addr":"172.21.15.143:6811/3348750750","heartbeat_front_addr":"172.21.15.143:6810/3348750750","state":["exists","up"]},{"osd":5,"uuid":"d4673f5b-8a8e-4559-a2a6-dadcc95e28b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":689,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3538568827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6805","nonce":3538568827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6807","nonce":3538568827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3538568827}]},"public_addr":"172.21.15.143:6804/3538568827","cluster_addr":"172.21.15.143:6805/3538568827","heartbeat_back_addr":"172.21.15.143:6807/3538568827","heartbeat_front_addr":"172.21.15.143:6806/3538568827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.912822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.644997+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.809738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.543973+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.760941+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:49:56.643053+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.14","mappings":[{"from":3,"to":1}]},{"pgid":"19.1c","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":15,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":53,"snaps":[{"begin":11,"length":2}]},{"pool":81,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":82,"snaps":[{"begin":2,"length":1}]},{"pool":91,"snaps":[{"begin":4,"length":4}]},{"pool":95,"snaps":[{"begin":6,"length":2}]},{"pool":102,"snaps":[{"begin":2,"length":1}]},{"pool":107,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":120,"snaps":[{"begin":12,"length":1}]},{"pool":123,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":147,"snaps":[{"begin":14,"length":1}]},{"pool":157,"snaps":[{"begin":17,"length":1}]},{"pool":163,"snaps":[{"begin":19,"length":2}]},{"pool":177,"snaps":[{"begin":24,"length":1}]},{"pool":241,"snaps":[{"begin":2,"length":1}]},{"pool":242,"snaps":[{"begin":4,"length":1}]},{"pool":245,"snaps":[{"begin":6,"length":1}]},{"pool":249,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 10 librados: watch_flush enter 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 10 librados: watch_flush exit 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 20 client.114143.objecter shutdown clearing up homeless session... 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 10 client.114143.objecter successfully canceled tick 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 msgr2=0x7f3e90039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 0x7f3e90039420 secure :-1 s=READY pgs=3914 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0007fd0 tx=0x7f3ea0009040 comp rx=0 tx=0).stop 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 10 monclient: shutdown 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 msgr2=0x7f3eb0122e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:11.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3eb0122e50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00a1150 tx=0x7f3ea8012580 comp rx=0 tx=0).stop 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 shutdown_connections 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3e9c00ac20 0x7f3eb0122e50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3e90036f60 0x7f3e90039420 unknown :-1 s=CLOSED pgs=3914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e9c00a010 0x7f3eb0122910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 --2- 172.21.15.5:0/1814754020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e9c003e00 0x7f3eb01223d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 >> 172.21.15.5:0/1814754020 conn(0x7f3eb00a45d0 msgr2=0x7f3eb006ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 shutdown_connections 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 -- 172.21.15.5:0/1814754020 wait complete. 2024-09-16T11:08:11.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:11.081+0000 7f3eb54a8700 1 librados: shutdown 2024-09-16T11:08:12.100 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T11:08:12.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 10 monclient: monmap: 2024-09-16T11:08:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.229 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:12.227769+0000 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:12.227769+0000 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:12.230 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 1 Processor -- start 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 1 -- start start 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.221+0000 7f6e07fff700 10 monclient: init 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 0x7f6e080a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6e080a2cf0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 0x7f6e080a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6e080a30d0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e08135b90 0x7f6e08135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6e08135b90 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0809edf0 con 0x7f6e08135b90 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0809fb30 con 0x7f6e080a30d0 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0809fe10 con 0x7f6e080a2cf0 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient(hunting): authenticate will time out at 1950.804596s 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e08135b90 0x7f6e08135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e08135b90 0x7f6e08135fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59978/0 (socket says 172.21.15.5:59978) 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 -- 172.21.15.5:0/304038838 learned_addr learned my addr 172.21.15.5:0/304038838 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): get_auth_request con 0x7f6e08135b90 auth_method 0 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 0x7f6e080a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 10 monclient(hunting): get_auth_request con 0x7f6e080a2cf0 auth_method 0 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 1 --2- 172.21.15.5:0/304038838 >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 0x7f6e080a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 10 monclient(hunting): get_auth_request con 0x7f6e080a30d0 auth_method 0 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient(hunting): handle_auth_done global_id 74697 payload 274 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 -- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 msgr2=0x7f6e080a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 0x7f6e080a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 -- 172.21.15.5:0/304038838 >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 msgr2=0x7f6e080a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 --2- 172.21.15.5:0/304038838 >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 0x7f6e080a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 monclient: found mon.noname-a 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 -- 172.21.15.5:0/304038838 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080a3d40 con 0x7f6e08135b90 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e067fc700 1 --2- 172.21.15.5:0/304038838 >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 0x7f6e080a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:12.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e06ffd700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 0x7f6e080a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e077fe700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e08135b90 0x7f6e08135fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a93b0 tx=0x7f6e0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c02754bdabd30f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 1 -- 172.21.15.5:0/304038838 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000e070 con 0x7f6e08135b90 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: dump: 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.245 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 1 -- 172.21.15.5:0/304038838 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0000b9c0 con 0x7f6e08135b90 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 5 monclient: authenticate success, global_id 74697 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 1 -- 172.21.15.5:0/304038838 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000bd70 con 0x7f6e08135b90 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 4 set_mon_vals no callback set 2024-09-16T11:08:12.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e05ffb700 10 monclient: dump: 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 10 monclient: shutdown 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e08135b90 msgr2=0x7f6e08135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e08135b90 0x7f6e08135fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a93b0 tx=0x7f6e0000b330 comp rx=0 tx=0).stop 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- 172.21.15.5:0/304038838 shutdown_connections 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e08135b90 0x7f6e08135fe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- 172.21.15.5:0/304038838 >> v2:172.21.15.143:3300/0 conn(0x7f6e080a30d0 0x7f6e080a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 --2- 172.21.15.5:0/304038838 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e080a2cf0 0x7f6e080a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- 172.21.15.5:0/304038838 >> 172.21.15.5:0/304038838 conn(0x7f6e080a84d0 msgr2=0x7f6e080a88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- 172.21.15.5:0/304038838 shutdown_connections 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.225+0000 7f6e07fff700 1 -- 172.21.15.5:0/304038838 wait complete. 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: monmap: 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 librados: starting msgr at 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 librados: starting objecter 2024-09-16T11:08:12.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 Processor -- start 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- start start 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 librados: setting wanted keys 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 librados: calling monclient init 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: init 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e080a2cf0 0x7f6e08070490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6e080a2cf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e080a30d0 0x7f6e08133f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6e080a30d0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 0x7f6e081344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6e08135b90 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08121090 con 0x7f6e080a30d0 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0811ffd0 con 0x7f6e080a2cf0 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08134a70 con 0x7f6e08135b90 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient(hunting): authenticate will time out at 1950.809792s 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 0x7f6e081344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e080a30d0 0x7f6e08133f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 0x7f6e081344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40142/0 (socket says 172.21.15.5:40142) 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 -- 172.21.15.5:0/1745289345 learned_addr learned my addr 172.21.15.5:0/1745289345 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): get_auth_request con 0x7f6e08135b90 auth_method 0 2024-09-16T11:08:12.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): get_auth_request con 0x7f6e080a30d0 auth_method 0 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e080a2cf0 0x7f6e08070490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): get_auth_request con 0x7f6e080a2cf0 auth_method 0 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e06ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient(hunting): handle_auth_done global_id 114149 payload 824 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 -- 172.21.15.5:0/1745289345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e080a2cf0 msgr2=0x7f6e08070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e080a2cf0 0x7f6e08070490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 -- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e080a30d0 msgr2=0x7f6e08133f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e080a30d0 0x7f6e08133f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 monclient: found mon.noname-c 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 -- 172.21.15.5:0/1745289345 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00009b30 con 0x7f6e08135b90 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e067fc700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e080a30d0 0x7f6e08133f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e077fe700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 0x7f6e081344b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a5e50 tx=0x7f6e00004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 client.?.objecter ms_handle_connect 0x7f6e08135b90 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00004040 con 0x7f6e08135b90 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: dump: 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 5 monclient: authenticate success, global_id 114149 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e000041c0 con 0x7f6e08135b90 2024-09-16T11:08:12.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _renew_subs 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e080a3b70 con 0x7f6e08135b90 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00004570 con 0x7f6e08135b90 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e057fa700 4 set_mon_vals no callback set 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 client.114149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6def7fe700 10 monclient: dump: 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.257 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _renew_subs 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08121930 con 0x7f6e08135b90 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.229+0000 7f6e07fff700 1 librados: init done 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6e000046d0 con 0x7f6e08135b90 2024-09-16T11:08:12.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 10 client.114149.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 15 client.114149.objecter _session_command_op_assign -1 1 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 20 client.114149.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 10 client.114149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e08134cd0 con 0x7f6e08135b90 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6df4036f60 0x7f6df4039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f6e00048e70 con 0x7f6e08135b90 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter ms_dispatch 0x7f6e08000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 3 client.114149.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 3 client.114149.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter checking command 1 2024-09-16T11:08:12.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6df403e060 0x7f6df4040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter _get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e06ffd700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6df4036f60 0x7f6df4039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 15 client.114149.objecter _session_command_op_remove -1 1 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter _get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 15 client.114149.objecter _session_command_op_assign 0 1 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 4 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter _send_command 1 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 --> v2:172.21.15.5:6808/2408116882 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6df4040b90 con 0x7f6df403e060 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter dump_active .. 0 homeless 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e06ffd700 10 monclient: get_auth_request con 0x7f6df4036f60 auth_method 0 2024-09-16T11:08:12.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e0002a060 con 0x7f6e08135b90 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e067fc700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6df403e060 0x7f6df4040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e067fc700 10 monclient: get_auth_request con 0x7f6df403e060 auth_method 0 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e067fc700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6df403e060 0x7f6df4040500 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter ms_handle_connect 0x7f6df403e060 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter ms_handle_connect 0x7f6df4036f60 2024-09-16T11:08:12.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6e06ffd700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6df4036f60 0x7f6df4039420 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 crypto rx=0x7f6e08070c80 tx=0x7f6df8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== osd.0 v2:172.21.15.5:6808/2408116882 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f6df4040b90 con 0x7f6df403e060 2024-09-16T11:08:12.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter ms_dispatch 0x7f6e08000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:12.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 10 client.114149.objecter _finish_command 1 = system:0 2024-09-16T11:08:12.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 4 2024-09-16T11:08:12.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.233+0000 7f6def7fe700 15 client.114149.objecter _session_command_op_remove 0 1 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 client.114149.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 15 client.114149.objecter _session_command_op_assign -1 2 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter _get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter _get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 15 client.114149.objecter _session_command_op_remove -1 2 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter get_session s=0x7f6df403d9b0 osd=0 3 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 15 client.114149.objecter _session_command_op_assign 0 2 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 4 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 client.114149.objecter _send_command 2 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 --> v2:172.21.15.5:6808/2408116882 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6e080a39f0 con 0x7f6df403e060 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6def7fe700 1 -- 172.21.15.5:0/1745289345 <== osd.0 v2:172.21.15.5:6808/2408116882 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6dfc009970 con 0x7f6df403e060 2024-09-16T11:08:12.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6def7fe700 10 client.114149.objecter ms_dispatch 0x7f6e08000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6def7fe700 10 client.114149.objecter _finish_command 2 = system:0 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6def7fe700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 4 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6def7fe700 15 client.114149.objecter _session_command_op_remove 0 2 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:12.264 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:12.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 librados: watch_flush enter 2024-09-16T11:08:12.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 librados: watch_flush exit 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 client.114149.objecter close_session for osd.0 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6df403e060 msgr2=0x7f6df4040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.5:6808/2408116882 conn(0x7f6df403e060 0x7f6df4040500 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter put_session s=0x7f6df403d9b0 osd=0 1 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 client.114149.objecter shutdown clearing up homeless session... 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 client.114149.objecter successfully canceled tick 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6df4036f60 msgr2=0x7f6df4039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 --2- 172.21.15.5:0/1745289345 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6df4036f60 0x7f6df4039420 secure :-1 s=READY pgs=3915 cs=0 l=1 rev1=1 crypto rx=0x7f6e08070c80 tx=0x7f6df8008040 comp rx=0 tx=0).stop 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 10 monclient: shutdown 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 msgr2=0x7f6e081344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 --2- 172.21.15.5:0/1745289345 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e08135b90 0x7f6e081344b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6e080a5e50 tx=0x7f6e00004af0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e077fe700 1 -- 172.21.15.5:0/1745289345 reap_dead start 2024-09-16T11:08:12.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 shutdown_connections 2024-09-16T11:08:12.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 >> 172.21.15.5:0/1745289345 conn(0x7f6e080a84d0 msgr2=0x7f6e08134f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 shutdown_connections 2024-09-16T11:08:12.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 -- 172.21.15.5:0/1745289345 wait complete. 2024-09-16T11:08:12.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.253+0000 7f6e07fff700 1 librados: shutdown 2024-09-16T11:08:12.273 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T11:08:12.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: get_monmap_and_config 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: monmap: 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:12.397718+0000 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:12.397718+0000 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:12.400 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:12.401 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 Processor -- start 2024-09-16T11:08:12.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 -- start start 2024-09-16T11:08:12.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: init 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88a809e8e0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-b con 0x7f88a809d6a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88a809dfd0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a8070e40 con 0x7f88a809dfd0 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a809fef0 con 0x7f88a809e8e0 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80a01d0 con 0x7f88a809d6a0 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient(hunting): authenticate will time out at 1950.974125s 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40148/0 (socket says 172.21.15.5:40148) 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 -- 172.21.15.5:0/3480001130 learned_addr learned my addr 172.21.15.5:0/3480001130 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): get_auth_request con 0x7f88a809d6a0 auth_method 0 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ada33700 1 --1- 172.21.15.5:0/3480001130 >> v1:172.21.15.5:6789/0 conn(0x7f88a809dfd0 0x7f88a813b0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51332/0 (socket says 172.21.15.5:51332) 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient(hunting): handle_auth_done global_id 114152 payload 274 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 -- 172.21.15.5:0/3480001130 >> v1:172.21.15.143:6789/0 conn(0x7f88a809e8e0 legacy=0x7f88a809ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 -- 172.21.15.5:0/3480001130 >> v1:172.21.15.5:6789/0 conn(0x7f88a809dfd0 legacy=0x7f88a813b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 monclient: found mon.noname-b 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 -- 172.21.15.5:0/3480001130 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a809f9f0 con 0x7f88a809d6a0 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88aca31700 1 --2- 172.21.15.5:0/3480001130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009e80 tx=0x7f88a400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fb95cf8c63ccea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 1 -- 172.21.15.5:0/3480001130 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400e070 con 0x7f88a809d6a0 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:12.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: dump: 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 1 -- 172.21.15.5:0/3480001130 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a400b9c0 con 0x7f88a809d6a0 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 5 monclient: authenticate success, global_id 114152 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 4 set_mon_vals no callback set 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 1 -- 172.21.15.5:0/3480001130 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a400bd70 con 0x7f88a809d6a0 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f889ffff700 10 monclient: dump: 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.413 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 10 monclient: shutdown 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 -- 172.21.15.5:0/3480001130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 msgr2=0x7f88a8138b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.393+0000 7f88ae234700 1 --2- 172.21.15.5:0/3480001130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009e80 tx=0x7f88a400b330 comp rx=0 tx=0).stop 2024-09-16T11:08:12.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- 172.21.15.5:0/3480001130 shutdown_connections 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 --2- 172.21.15.5:0/3480001130 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809d6a0 0x7f88a8138b80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- 172.21.15.5:0/3480001130 >> 172.21.15.5:0/3480001130 conn(0x7f88a80a45d0 msgr2=0x7f88a80a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- 172.21.15.5:0/3480001130 shutdown_connections 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- 172.21.15.5:0/3480001130 wait complete. 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient: monmap: 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.415 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 librados: starting msgr at 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 librados: starting objecter 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 Processor -- start 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- start start 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 librados: setting wanted keys 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 librados: calling monclient init 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient: init 2024-09-16T11:08:12.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 0x7f88a8133a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-a con 0x7f88a809d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a809dfd0 0x7f88a8133f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-b con 0x7f88a809dfd0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): picked mon.noname-c con 0x7f88a809e8e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a809b0d0 con 0x7f88a809d6a0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a806fa10 con 0x7f88a809dfd0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a806ceb0 con 0x7f88a809e8e0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ae234700 10 monclient(hunting): authenticate will time out at 1950.978607s 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 0x7f88a8133a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 0x7f88a8133a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59994/0 (socket says 172.21.15.5:59994) 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:40160/0 (socket says 172.21.15.5:40160) 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 1 -- 172.21.15.5:0/1109554320 learned_addr learned my addr 172.21.15.5:0/1109554320 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): get_auth_request con 0x7f88a809d6a0 auth_method 0 2024-09-16T11:08:12.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): get_auth_request con 0x7f88a809e8e0 auth_method 0 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a809dfd0 0x7f88a8133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): get_auth_request con 0x7f88a809dfd0 auth_method 0 2024-09-16T11:08:12.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ad232700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88aca31700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient(hunting): handle_auth_done global_id 114155 payload 824 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 -- 172.21.15.5:0/1109554320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a809dfd0 msgr2=0x7f88a8133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a809dfd0 0x7f88a8133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 -- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 msgr2=0x7f88a8133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 0x7f88a8133a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 monclient: found mon.noname-c 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 -- 172.21.15.5:0/1109554320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4009b30 con 0x7f88a809e8e0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f88ada33700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c400 tx=0x7f88a000c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f889dffb700 10 client.?.objecter ms_handle_connect 0x7f88a809e8e0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f889dffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.397+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0004a90 con 0x7f88a809e8e0 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: dump: 2024-09-16T11:08:12.421 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 5 monclient: authenticate success, global_id 114155 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0004bf0 con 0x7f88a809e8e0 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _renew_subs 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a8120e50 con 0x7f88a809e8e0 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a00070d0 con 0x7f88a809e8e0 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889f7fe700 4 set_mon_vals no callback set 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 monclient: dump: 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.423 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 client.114155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _renew_subs 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a8121410 con 0x7f88a809e8e0 2024-09-16T11:08:12.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 1 librados: init done 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f88a0007230 con 0x7f88a809e8e0 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a809ecc0 con 0x7f88a809e8e0 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 1 --2- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 0x7f888c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f88a0047200 con 0x7f88a809e8e0 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 client.114155.objecter ms_dispatch 0x7f88a8000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 3 client.114155.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 3 client.114155.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 20 client.114155.objecter dump_active .. 0 homeless 2024-09-16T11:08:12.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ad232700 1 --2- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 0x7f888c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ad232700 10 monclient: get_auth_request con 0x7f888c036f60 auth_method 0 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f88ad232700 1 --2- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 0x7f888c039420 secure :-1 s=READY pgs=3916 cs=0 l=1 rev1=1 crypto rx=0x7f8898001940 tx=0x7f8898009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.401+0000 7f889dffb700 10 client.114155.objecter ms_handle_connect 0x7f888c036f60 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.409+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f88a00074e0 con 0x7f88a809e8e0 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.409+0000 7f889dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.409+0000 7f889dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:12.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f88a809ad80 con 0x7f888c036f60 2024-09-16T11:08:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f889dffb700 1 -- 172.21.15.5:0/1109554320 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f88980015e0 con 0x7f888c036f60 2024-09-16T11:08:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 10 librados: watch_flush enter 2024-09-16T11:08:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 10 librados: watch_flush exit 2024-09-16T11:08:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 20 client.114155.objecter shutdown clearing up homeless session... 2024-09-16T11:08:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 10 client.114155.objecter successfully canceled tick 2024-09-16T11:08:12.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 msgr2=0x7f888c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 0x7f888c039420 secure :-1 s=READY pgs=3916 cs=0 l=1 rev1=1 crypto rx=0x7f8898001940 tx=0x7f8898009040 comp rx=0 tx=0).stop 2024-09-16T11:08:12.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 10 monclient: shutdown 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 msgr2=0x7f88a811d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c400 tx=0x7f88a000c710 comp rx=0 tx=0).stop 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 shutdown_connections 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f88a809e8e0 0x7f88a811d550 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88a809dfd0 0x7f88a8133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> v2:172.21.15.143:6812/1098205034 conn(0x7f888c036f60 0x7f888c039420 unknown :-1 s=CLOSED pgs=3916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 --2- 172.21.15.5:0/1109554320 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88a809d6a0 0x7f88a8133a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.585+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 >> 172.21.15.5:0/1109554320 conn(0x7f88a80a45d0 msgr2=0x7f88a80a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.589+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 shutdown_connections 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.589+0000 7f88ae234700 1 -- 172.21.15.5:0/1109554320 wait complete. 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.589+0000 7f88ae234700 1 librados: shutdown 2024-09-16T11:08:12.599 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T11:08:12.609 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T11:08:12.610 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: get_monmap_and_config 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: monmap: 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:12.730429+0000 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:12.730429+0000 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.732 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:12.733 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 Processor -- start 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 -- start start 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: init 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T11:08:12.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe90009da50 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe900099f10 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 0x7fe90013c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe90013a4f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90009de30 con 0x7fe900099f10 2024-09-16T11:08:12.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000a0b70 con 0x7fe90009da50 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000985d0 con 0x7fe90013a4f0 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe905a3c700 10 monclient(hunting): authenticate will time out at 1951.307278s 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fe7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe900099f10 0x7fe9000a0460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34554/0 (socket says 172.21.15.5:34554) 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fe7fc700 1 -- 172.21.15.5:0/1173664420 learned_addr learned my addr 172.21.15.5:0/1173664420 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 1 --2- 172.21.15.5:0/1173664420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 0x7fe90013c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): get_auth_request con 0x7fe90013a4f0 auth_method 0 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 1 -- 172.21.15.5:0/1173664420 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014615376 0 0) 0x7fe90009de30 con 0x7fe900099f10 2024-09-16T11:08:12.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 10 monclient(hunting): _negotiate 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 10 monclient(hunting): my global_id is 74709 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8fdffb700 1 -- 172.21.15.5:0/1173664420 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ec0035d0 con 0x7fe900099f10 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.725+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_done global_id 114158 payload 274 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 -- 172.21.15.5:0/1173664420 >> v1:172.21.15.143:6789/0 conn(0x7fe90009da50 legacy=0x7fe90009fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 -- 172.21.15.5:0/1173664420 >> v1:172.21.15.5:6789/0 conn(0x7fe900099f10 legacy=0x7fe9000a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 monclient: found mon.noname-b 2024-09-16T11:08:12.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 -- 172.21.15.5:0/1173664420 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900099af0 con 0x7fe90013a4f0 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 --2- 172.21.15.5:0/1173664420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 0x7fe90013c8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009dc0 tx=0x7fe8f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f0dc63e866ef350 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 1 -- 172.21.15.5:0/1173664420 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f800e070 con 0x7fe90013a4f0 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: dump: 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.746 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 1 -- 172.21.15.5:0/1173664420 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f800bb20 con 0x7fe90013a4f0 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 5 monclient: authenticate success, global_id 114158 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:12.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 1 -- 172.21.15.5:0/1173664420 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8012430 con 0x7fe90013a4f0 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 4 set_mon_vals no callback set 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fdffb700 10 monclient: dump: 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.748 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: shutdown 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- 172.21.15.5:0/1173664420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 msgr2=0x7fe90013c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 --2- 172.21.15.5:0/1173664420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 0x7fe90013c8e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009dc0 tx=0x7fe8f800b330 comp rx=0 tx=0).stop 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- 172.21.15.5:0/1173664420 shutdown_connections 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 --2- 172.21.15.5:0/1173664420 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe90013a4f0 0x7fe90013c8e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- 172.21.15.5:0/1173664420 >> 172.21.15.5:0/1173664420 conn(0x7fe9000a0e40 msgr2=0x7fe9000a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- 172.21.15.5:0/1173664420 shutdown_connections 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- 172.21.15.5:0/1173664420 wait complete. 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: monmap: 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:12.750 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 librados: starting msgr at 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 librados: starting objecter 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 Processor -- start 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- start start 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 librados: setting wanted keys 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 librados: calling monclient init 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: init 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 0x7fe9001cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-c con 0x7fe900099f10 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe90009da50 0x7fe9001cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe90009da50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe90013a4f0 0x7fe90006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): picked mon.noname-a con 0x7fe90013a4f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90013c960 con 0x7fe90013a4f0 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90009b530 con 0x7fe90009da50 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe900134380 con 0x7fe900099f10 2024-09-16T11:08:12.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe905a3c700 10 monclient(hunting): authenticate will time out at 1951.311194s 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 0x7fe9001cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 0x7fe9001cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52938/0 (socket says 172.21.15.5:52938) 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 -- 172.21.15.5:0/3531477360 learned_addr learned my addr 172.21.15.5:0/3531477360 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): get_auth_request con 0x7fe900099f10 auth_method 0 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe90013a4f0 0x7fe90006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): get_auth_request con 0x7fe90013a4f0 auth_method 0 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fe7fc700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe90009da50 0x7fe9001cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fe7fc700 10 monclient(hunting): get_auth_request con 0x7fe90009da50 auth_method 0 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fe7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8fe7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient(hunting): handle_auth_done global_id 114161 payload 824 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 -- 172.21.15.5:0/3531477360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe90009da50 msgr2=0x7fe9001cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe90009da50 0x7fe9001cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 -- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe90013a4f0 msgr2=0x7fe90006e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe90013a4f0 0x7fe90006e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 monclient: found mon.noname-c 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.729+0000 7fe8feffd700 1 -- 172.21.15.5:0/3531477360 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f8009a70 con 0x7fe900099f10 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8feffd700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 0x7fe9001cb200 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c500 tx=0x7fe8f000c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.?.objecter ms_handle_connect 0x7fe900099f10 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f00052d0 con 0x7fe900099f10 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: dump: 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 5 monclient: authenticate success, global_id 114161 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0005430 con 0x7fe900099f10 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 monclient: _renew_subs 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe900134660 con 0x7fe900099f10 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f00055a0 con 0x7fe900099f10 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 client.114161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: dump: 2024-09-16T11:08:12.757 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8fd7fa700 4 set_mon_vals no callback set 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 monclient: _renew_subs 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9001cbc80 con 0x7fe900099f10 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 1 librados: init done 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 client.114161.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 15 client.114161.objecter _session_command_op_assign -1 1 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 20 client.114161.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 client.114161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe90009e130 con 0x7fe900099f10 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7fe8f00038e0 con 0x7fe900099f10 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe8e0036f60 0x7fe8e0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7fe8f0046ff0 con 0x7fe900099f10 2024-09-16T11:08:12.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.114161.objecter ms_dispatch 0x7fe900000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 3 client.114161.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 3 client.114161.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.114161.objecter checking command 1 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.5:6800/3675136489 conn(0x7fe8e003e060 0x7fe8e0040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter _get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 15 client.114161.objecter _session_command_op_remove -1 1 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter _get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 15 client.114161.objecter _session_command_op_assign 1 1 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 4 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.114161.objecter _send_command 1 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 --> v2:172.21.15.5:6800/3675136489 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8e0040b90 con 0x7fe8e003e060 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 20 client.114161.objecter dump_active .. 0 homeless 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8f00495f0 con 0x7fe900099f10 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8fe7fc700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe8e0036f60 0x7fe8e0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8ff7fe700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.5:6800/3675136489 conn(0x7fe8e003e060 0x7fe8e0040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8fe7fc700 10 monclient: get_auth_request con 0x7fe8e0036f60 auth_method 0 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8ff7fe700 10 monclient: get_auth_request con 0x7fe8e003e060 auth_method 0 2024-09-16T11:08:12.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8ff7fe700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.5:6800/3675136489 conn(0x7fe8e003e060 0x7fe8e0040500 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.114161.objecter ms_handle_connect 0x7fe8e003e060 2024-09-16T11:08:12.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8f7fff700 10 client.114161.objecter ms_handle_connect 0x7fe8e0036f60 2024-09-16T11:08:12.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.733+0000 7fe8fe7fc700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe8e0036f60 0x7fe8e0039420 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7fe900070f10 tx=0x7fe8e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.737+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== osd.1 v2:172.21.15.5:6800/3675136489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fe8e0040b90 con 0x7fe8e003e060 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.737+0000 7fe8f7fff700 10 client.114161.objecter ms_dispatch 0x7fe900000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.737+0000 7fe8f7fff700 10 client.114161.objecter _finish_command 1 = system:0 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.737+0000 7fe8f7fff700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 4 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.737+0000 7fe8f7fff700 15 client.114161.objecter _session_command_op_remove 1 1 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 client.114161.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 15 client.114161.objecter _session_command_op_assign -1 2 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter _get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter _get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 15 client.114161.objecter _session_command_op_remove -1 2 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter get_session s=0x7fe8e003d9b0 osd=1 3 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 15 client.114161.objecter _session_command_op_assign 1 2 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 4 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 client.114161.objecter _send_command 2 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 --> v2:172.21.15.5:6800/3675136489 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe900099810 con 0x7fe8e003e060 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8f7fff700 1 -- 172.21.15.5:0/3531477360 <== osd.1 v2:172.21.15.5:6800/3675136489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8f8003d00 con 0x7fe8e003e060 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8f7fff700 10 client.114161.objecter ms_dispatch 0x7fe900000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8f7fff700 10 client.114161.objecter _finish_command 2 = system:0 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8f7fff700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 4 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8f7fff700 15 client.114161.objecter _session_command_op_remove 1 2 2024-09-16T11:08:12.764 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:12.765 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:12.765 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 librados: watch_flush enter 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 librados: watch_flush exit 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 client.114161.objecter close_session for osd.1 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 >> v2:172.21.15.5:6800/3675136489 conn(0x7fe8e003e060 msgr2=0x7fe8e0040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.5:6800/3675136489 conn(0x7fe8e003e060 0x7fe8e0040500 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter put_session s=0x7fe8e003d9b0 osd=1 1 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 client.114161.objecter shutdown clearing up homeless session... 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 client.114161.objecter successfully canceled tick 2024-09-16T11:08:12.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe8e0036f60 msgr2=0x7fe8e0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 --2- 172.21.15.5:0/3531477360 >> v2:172.21.15.143:6812/1098205034 conn(0x7fe8e0036f60 0x7fe8e0039420 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7fe900070f10 tx=0x7fe8e8009040 comp rx=0 tx=0).stop 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 10 monclient: shutdown 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 msgr2=0x7fe9001cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 --2- 172.21.15.5:0/3531477360 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7fe900099f10 0x7fe9001cb200 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c500 tx=0x7fe8f000c810 comp rx=0 tx=0).stop 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe8ff7fe700 1 -- 172.21.15.5:0/3531477360 reap_dead start 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 shutdown_connections 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 >> 172.21.15.5:0/3531477360 conn(0x7fe9000a0e40 msgr2=0x7fe9000a2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 shutdown_connections 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 -- 172.21.15.5:0/3531477360 wait complete. 2024-09-16T11:08:12.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.753+0000 7fe905a3c700 1 librados: shutdown 2024-09-16T11:08:12.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T11:08:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:12.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 10 monclient: monmap: 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:12.887162+0000 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:12.887162+0000 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:12.889 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:12.890 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 1 Processor -- start 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 1 -- start start 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.881+0000 7f6e83fff700 10 monclient: init 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e840a2cf0 0x7f6e840a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6e840a2cf0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e840a3e90 0x7f6e840a3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6e840a3e90 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6e84135ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840a4a80 con 0x7f6e840a3e90 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840a3b50 con 0x7f6e840a2cf0 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840a4310 con 0x7f6e84135ab0 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient(hunting): authenticate will time out at 1951.464437s 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e840a3e90 0x7f6e840a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84135ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52954/0 (socket says 172.21.15.5:52954) 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 1 -- 172.21.15.5:0/1753859712 learned_addr learned my addr 172.21.15.5:0/1753859712 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): get_auth_request con 0x7f6e84135ab0 auth_method 0 2024-09-16T11:08:12.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/1753859712 >> v2:172.21.15.143:3300/0 conn(0x7f6e840a2cf0 0x7f6e840a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): get_auth_request con 0x7f6e840a3e90 auth_method 0 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): get_auth_request con 0x7f6e840a2cf0 auth_method 0 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_done global_id 94393 payload 274 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 -- 172.21.15.5:0/1753859712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 msgr2=0x7f6e84135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/1753859712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84135ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 -- 172.21.15.5:0/1753859712 >> v2:172.21.15.5:3300/0 conn(0x7f6e840a3e90 msgr2=0x7f6e840a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/1753859712 >> v2:172.21.15.5:3300/0 conn(0x7f6e840a3e90 0x7f6e840a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 monclient: found mon.noname-c 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 -- 172.21.15.5:0/1753859712 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e840a06c0 con 0x7f6e840a2cf0 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/1753859712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e840a30d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e840a5b30 tx=0x7f6e74009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c26d0e06030147 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 1 -- 172.21.15.5:0/1753859712 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74011830 con 0x7f6e840a2cf0 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: dump: 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.897 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 5 monclient: authenticate success, global_id 94393 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 1 -- 172.21.15.5:0/1753859712 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e74011990 con 0x7f6e840a2cf0 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 4 set_mon_vals no callback set 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 1 -- 172.21.15.5:0/1753859712 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74011d20 con 0x7f6e840a2cf0 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e81ffb700 10 monclient: dump: 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.899 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.900 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 10 monclient: shutdown 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- 172.21.15.5:0/1753859712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 msgr2=0x7f6e840a30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- 172.21.15.5:0/1753859712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e840a30d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e840a5b30 tx=0x7f6e74009b30 comp rx=0 tx=0).stop 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- 172.21.15.5:0/1753859712 shutdown_connections 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- 172.21.15.5:0/1753859712 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84135ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- 172.21.15.5:0/1753859712 >> v2:172.21.15.5:3300/0 conn(0x7f6e840a3e90 0x7f6e840a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 --2- 172.21.15.5:0/1753859712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e840a30d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- 172.21.15.5:0/1753859712 >> 172.21.15.5:0/1753859712 conn(0x7f6e840a89e0 msgr2=0x7f6e840a4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- 172.21.15.5:0/1753859712 shutdown_connections 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.885+0000 7f6e83fff700 1 -- 172.21.15.5:0/1753859712 wait complete. 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient: monmap: 2024-09-16T11:08:12.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 librados: starting msgr at 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 librados: starting objecter 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 Processor -- start 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 -- start start 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 librados: setting wanted keys 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 librados: calling monclient init 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient: init 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e84070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6e840a2cf0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 0x7f6e84133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6e840a3e90 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:12.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6e84135ab0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840a66f0 con 0x7f6e840a3e90 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8409f550 con 0x7f6e840a2cf0 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84121410 con 0x7f6e84135ab0 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e83fff700 10 monclient(hunting): authenticate will time out at 1951.470900s 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 0x7f6e84133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84134250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52966/0 (socket says 172.21.15.5:52966) 2024-09-16T11:08:12.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 1 -- 172.21.15.5:0/557137874 learned_addr learned my addr 172.21.15.5:0/557137874 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): get_auth_request con 0x7f6e84135ab0 auth_method 0 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): get_auth_request con 0x7f6e840a3e90 auth_method 0 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e84070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): get_auth_request con 0x7f6e840a2cf0 auth_method 0 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:12.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e82ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient(hunting): handle_auth_done global_id 74718 payload 824 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 -- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 msgr2=0x7f6e84134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84134250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 -- 172.21.15.5:0/557137874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 msgr2=0x7f6e84070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e84070270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient: _finish_hunting 0 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 monclient: found mon.noname-a 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 -- 172.21.15.5:0/557137874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e740094b0 con 0x7f6e840a3e90 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e827fc700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 0x7f6e84133d10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800e8d0 tx=0x7f6e7800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e6b7fe700 10 client.?.objecter ms_handle_connect 0x7f6e840a3e90 2024-09-16T11:08:12.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.889+0000 7f6e6b7fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78013680 con 0x7f6e840a3e90 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: dump: 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.914 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e780137e0 con 0x7f6e840a3e90 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78013b50 con 0x7f6e840a3e90 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 5 monclient: authenticate success, global_id 74718 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _renew_subs 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e817fa700 4 set_mon_vals no callback set 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84120e50 con 0x7f6e840a3e90 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 monclient: dump: 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 client.74718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _renew_subs 2024-09-16T11:08:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e84070eb0 con 0x7f6e840a3e90 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 1 librados: init done 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84134a00 con 0x7f6e840a3e90 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6e78013cb0 con 0x7f6e840a3e90 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 --2- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 0x7f6e70039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f6e780479d0 con 0x7f6e840a3e90 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 client.74718.objecter ms_dispatch 0x7f6e84000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 3 client.74718.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 3 client.74718.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 20 client.74718.objecter dump_active .. 0 homeless 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 0x7f6e70039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e82ffd700 10 monclient: get_auth_request con 0x7f6e70036f90 auth_method 0 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e6b7fe700 10 client.74718.objecter ms_handle_connect 0x7f6e70036f90 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.893+0000 7f6e82ffd700 1 --2- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 0x7f6e70039450 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:12.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.901+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6e7801bbe0 con 0x7f6e840a3e90 2024-09-16T11:08:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.901+0000 7f6e6b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:12.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:12.901+0000 7f6e6b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.069+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e840a2660 con 0x7f6e70036f90 2024-09-16T11:08:13.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e6b7fe700 1 -- 172.21.15.5:0/557137874 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6e74011630 con 0x7f6e70036f90 2024-09-16T11:08:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 10 librados: watch_flush enter 2024-09-16T11:08:13.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 10 librados: watch_flush exit 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 20 client.74718.objecter shutdown clearing up homeless session... 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 10 client.74718.objecter successfully canceled tick 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 msgr2=0x7f6e70039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 0x7f6e70039450 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74008030 comp rx=0 tx=0).stop 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 10 monclient: shutdown 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 msgr2=0x7f6e84133d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 0x7f6e84133d10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800e8d0 tx=0x7f6e7800ebe0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 shutdown_connections 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6e84135ab0 0x7f6e84134250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6e70036f90 0x7f6e70039450 unknown :-1 s=CLOSED pgs=3918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e840a3e90 0x7f6e84133d10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 --2- 172.21.15.5:0/557137874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e840a2cf0 0x7f6e84070270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 >> 172.21.15.5:0/557137874 conn(0x7f6e840a89e0 msgr2=0x7f6e84134f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 shutdown_connections 2024-09-16T11:08:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 -- 172.21.15.5:0/557137874 wait complete. 2024-09-16T11:08:13.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.073+0000 7f6e83fff700 1 librados: shutdown 2024-09-16T11:08:13.085 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T11:08:13.093 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T11:08:13.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:13.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.201+0000 7f3d7f270700 10 monclient: get_monmap_and_config 2024-09-16T11:08:13.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.201+0000 7f3d7f270700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.201+0000 7f3d7f270700 10 monclient: monmap: 2024-09-16T11:08:13.210 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.210 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:13.208450+0000 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:13.208450+0000 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:13.211 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 Processor -- start 2024-09-16T11:08:13.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 -- start start 2024-09-16T11:08:13.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient: init 2024-09-16T11:08:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T11:08:13.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d780a2cf0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d7813af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-b con 0x7f3d780a37e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3d7813b440 0x7f3d7813d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-a con 0x7f3d7813b440 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d780a3e20 con 0x7f3d780a2cf0 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7809f120 con 0x7f3d7813b440 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7809f920 con 0x7f3d780a37e0 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient(hunting): authenticate will time out at 1951.784929s 2024-09-16T11:08:13.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7e26e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3d780a2cf0 0x7f3d780a30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34562/0 (socket says 172.21.15.5:34562) 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7e26e700 1 -- 172.21.15.5:0/1079182306 learned_addr learned my addr 172.21.15.5:0/1079182306 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/1079182306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d7813af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 1 --2- 172.21.15.5:0/1079182306 >> v2:172.21.15.5:3300/0 conn(0x7f3d7813b440 0x7f3d7813d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): get_auth_request con 0x7f3d780a37e0 auth_method 0 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): get_auth_request con 0x7f3d7813b440 auth_method 0 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7ea6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_done global_id 114170 payload 274 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 -- 172.21.15.5:0/1079182306 >> v2:172.21.15.5:3300/0 conn(0x7f3d7813b440 msgr2=0x7f3d7813d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/1079182306 >> v2:172.21.15.5:3300/0 conn(0x7f3d7813b440 0x7f3d7813d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 -- 172.21.15.5:0/1079182306 >> v1:172.21.15.5:6789/0 conn(0x7f3d780a2cf0 legacy=0x7f3d780a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 monclient: found mon.noname-b 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 -- 172.21.15.5:0/1079182306 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7809fc00 con 0x7f3d780a37e0 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/1079182306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d7813af00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009850 tx=0x7f3d74009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42864453efc7c578 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 1 -- 172.21.15.5:0/1079182306 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74004230 con 0x7f3d780a37e0 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: dump: 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.219 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 1 -- 172.21.15.5:0/1079182306 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d74004390 con 0x7f3d780a37e0 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 5 monclient: authenticate success, global_id 114170 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 1 -- 172.21.15.5:0/1079182306 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d740036c0 con 0x7f3d780a37e0 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7d26c700 10 monclient: dump: 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.221 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 4 set_mon_vals no callback set 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 10 monclient: shutdown 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 -- 172.21.15.5:0/1079182306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 msgr2=0x7f3d7813af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.205+0000 7f3d7f270700 1 --2- 172.21.15.5:0/1079182306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d7813af00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009850 tx=0x7f3d74009b60 comp rx=0 tx=0).stop 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 -- 172.21.15.5:0/1079182306 shutdown_connections 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 --2- 172.21.15.5:0/1079182306 >> v2:172.21.15.5:3300/0 conn(0x7f3d7813b440 0x7f3d7813d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 --2- 172.21.15.5:0/1079182306 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d7813af00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 -- 172.21.15.5:0/1079182306 >> 172.21.15.5:0/1079182306 conn(0x7f3d78073e00 msgr2=0x7f3d78074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 -- 172.21.15.5:0/1079182306 shutdown_connections 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.209+0000 7f3d7f270700 1 -- 172.21.15.5:0/1079182306 wait complete. 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: monmap: 2024-09-16T11:08:13.229 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 librados: starting msgr at 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 librados: starting objecter 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 Processor -- start 2024-09-16T11:08:13.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- start start 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 librados: setting wanted keys 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 librados: calling monclient init 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: init 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d780a2cf0 0x7f3d781ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-a con 0x7f3d780a2cf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d78135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-c con 0x7f3d780a37e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7813b440 0x7f3d78136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): picked mon.noname-b con 0x7f3d7813b440 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781cad90 con 0x7f3d780a2cf0 2024-09-16T11:08:13.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780a7650 con 0x7f3d7813b440 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780a7200 con 0x7f3d780a37e0 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient(hunting): authenticate will time out at 1951.794248s 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d78135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d780a2cf0 0x7f3d781ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d78135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:52994/0 (socket says 172.21.15.5:52994) 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 1 -- 172.21.15.5:0/3743217954 learned_addr learned my addr 172.21.15.5:0/3743217954 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): get_auth_request con 0x7f3d780a37e0 auth_method 0 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): get_auth_request con 0x7f3d780a2cf0 auth_method 0 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7813b440 0x7f3d78136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 10 monclient(hunting): get_auth_request con 0x7f3d7813b440 auth_method 0 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient(hunting): handle_auth_done global_id 74727 payload 824 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 -- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 msgr2=0x7f3d78135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d78135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 -- 172.21.15.5:0/3743217954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7813b440 msgr2=0x7f3d78136500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7813b440 0x7f3d78136500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 -- 172.21.15.5:0/3743217954 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740094b0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7ea6f700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d7813b440 0x7f3d78136500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3d780a37e0 0x7f3d78135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7e26e700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d780a2cf0 0x7f3d781ca850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c780 tx=0x7f3d6800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 client.?.objecter ms_handle_connect 0x7f3d780a2cf0 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68004ca0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: dump: 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.235 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68004e00 con 0x7f3d780a2cf0 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 5 monclient: authenticate success, global_id 74727 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _renew_subs 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78071630 con 0x7f3d780a2cf0 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d680072f0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6effd700 10 monclient: dump: 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.237 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 client.74727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _renew_subs 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d6ffff700 4 set_mon_vals no callback set 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d781226b0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.213+0000 7f3d7f270700 1 librados: init done 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 10 client.74727.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 15 client.74727.objecter _session_command_op_assign -1 1 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 20 client.74727.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:13.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 10 client.74727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3d780714b0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f3d68007450 con 0x7f3d780a2cf0 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d5c036f60 0x7f3d5c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f3d680480d0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 client.74727.objecter ms_dispatch 0x7f3d78000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 3 client.74727.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 3 client.74727.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 client.74727.objecter checking command 1 2024-09-16T11:08:13.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.5:6804/2015722671 conn(0x7f3d5c03e060 0x7f3d5c040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter _get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 15 client.74727.objecter _session_command_op_remove -1 1 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter _get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 15 client.74727.objecter _session_command_op_assign 2 1 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 4 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 client.74727.objecter _send_command 1 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 --> v2:172.21.15.5:6804/2015722671 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d5c040b90 con 0x7f3d5c03e060 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 20 client.74727.objecter dump_active .. 0 homeless 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d680484f0 con 0x7f3d780a2cf0 2024-09-16T11:08:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7ea6f700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.5:6804/2015722671 conn(0x7f3d5c03e060 0x7f3d5c040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d5c036f60 0x7f3d5c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7ea6f700 10 monclient: get_auth_request con 0x7f3d5c03e060 auth_method 0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7da6d700 10 monclient: get_auth_request con 0x7f3d5c036f60 auth_method 0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 client.74727.objecter ms_handle_connect 0x7f3d5c03e060 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7ea6f700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.5:6804/2015722671 conn(0x7f3d5c03e060 0x7f3d5c040500 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d6effd700 10 client.74727.objecter ms_handle_connect 0x7f3d5c036f60 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.217+0000 7f3d7da6d700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d5c036f60 0x7f3d5c039420 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f3d78071480 tx=0x7f3d74008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.221+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== osd.2 v2:172.21.15.5:6804/2015722671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3d5c040b90 con 0x7f3d5c03e060 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.221+0000 7f3d6effd700 10 client.74727.objecter ms_dispatch 0x7f3d78000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.221+0000 7f3d6effd700 10 client.74727.objecter _finish_command 1 = system:0 2024-09-16T11:08:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.221+0000 7f3d6effd700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 4 2024-09-16T11:08:13.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.221+0000 7f3d6effd700 15 client.74727.objecter _session_command_op_remove 2 1 2024-09-16T11:08:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 client.74727.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 15 client.74727.objecter _session_command_op_assign -1 2 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter _get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter _get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 15 client.74727.objecter _session_command_op_remove -1 2 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter get_session s=0x7f3d5c03d9b0 osd=2 3 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 15 client.74727.objecter _session_command_op_assign 2 2 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 4 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 client.74727.objecter _send_command 2 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 --> v2:172.21.15.5:6804/2015722671 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3d780a0ba0 con 0x7f3d5c03e060 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d6effd700 1 -- 172.21.15.5:0/3743217954 <== osd.2 v2:172.21.15.5:6804/2015722671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3d7000f040 con 0x7f3d5c03e060 2024-09-16T11:08:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d6effd700 10 client.74727.objecter ms_dispatch 0x7f3d78000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d6effd700 10 client.74727.objecter _finish_command 2 = system:0 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d6effd700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 4 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d6effd700 15 client.74727.objecter _session_command_op_remove 2 2 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:13.247 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 librados: watch_flush enter 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 librados: watch_flush exit 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 client.74727.objecter close_session for osd.2 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 >> v2:172.21.15.5:6804/2015722671 conn(0x7f3d5c03e060 msgr2=0x7f3d5c040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.5:6804/2015722671 conn(0x7f3d5c03e060 0x7f3d5c040500 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter put_session s=0x7f3d5c03d9b0 osd=2 1 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 client.74727.objecter shutdown clearing up homeless session... 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 client.74727.objecter successfully canceled tick 2024-09-16T11:08:13.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d5c036f60 msgr2=0x7f3d5c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 --2- 172.21.15.5:0/3743217954 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3d5c036f60 0x7f3d5c039420 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f3d78071480 tx=0x7f3d74008030 comp rx=0 tx=0).stop 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 10 monclient: shutdown 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d780a2cf0 msgr2=0x7f3d781ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7f270700 1 --2- 172.21.15.5:0/3743217954 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3d780a2cf0 0x7f3d781ca850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c780 tx=0x7f3d6800ca90 comp rx=0 tx=0).stop 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.237+0000 7f3d7ea6f700 1 -- 172.21.15.5:0/3743217954 reap_dead start 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.241+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 shutdown_connections 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.241+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 >> 172.21.15.5:0/3743217954 conn(0x7f3d78073e00 msgr2=0x7f3d780a6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.241+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 shutdown_connections 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.241+0000 7f3d7f270700 1 -- 172.21.15.5:0/3743217954 wait complete. 2024-09-16T11:08:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.241+0000 7f3d7f270700 1 librados: shutdown 2024-09-16T11:08:13.263 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 10 monclient: get_monmap_and_config 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 10 monclient: monmap: 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.377 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:13.375783+0000 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:13.375783+0000 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:13.378 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 1 Processor -- start 2024-09-16T11:08:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 1 -- start start 2024-09-16T11:08:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.369+0000 7f61c21c8700 10 monclient: init 2024-09-16T11:08:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61bc09d720 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61bc09db00 0x7f61bc09eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-a con 0x7f61bc09db00 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 0x7f61bc139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-c con 0x7f61bc137ab0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61bc09bc50 con 0x7f61bc09d720 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc1326f0 con 0x7f61bc09db00 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc09b970 con 0x7f61bc137ab0 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient(hunting): authenticate will time out at 1951.952278s 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61bc09db00 0x7f61bc09eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 0x7f61bc139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61bc09db00 0x7f61bc09eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51892/0 (socket says 172.21.15.5:51892) 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 -- 172.21.15.5:0/2631467129 learned_addr learned my addr 172.21.15.5:0/2631467129 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): get_auth_request con 0x7f61bc09db00 auth_method 0 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): get_auth_request con 0x7f61bc137ab0 auth_method 0 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient(hunting): handle_auth_done global_id 74730 payload 274 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 -- 172.21.15.5:0/2631467129 >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 msgr2=0x7f61bc139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 --2- 172.21.15.5:0/2631467129 >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 0x7f61bc139ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 -- 172.21.15.5:0/2631467129 >> v1:172.21.15.143:6789/0 conn(0x7f61bc09d720 legacy=0x7f61bc09e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 -- 172.21.15.5:0/2631467129 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc09bf30 con 0x7f61bc09db00 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c19c7700 1 --2- 172.21.15.5:0/2631467129 >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 0x7f61bc139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c09c5700 1 --2- 172.21.15.5:0/2631467129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09db00 0x7f61bc09eff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0a1a70 tx=0x7f61b800d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7f6466c3373cd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 1 -- 172.21.15.5:0/2631467129 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b8010070 con 0x7f61bc09db00 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: dump: 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.391 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 5 monclient: authenticate success, global_id 74730 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 1 -- 172.21.15.5:0/2631467129 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b800d9d0 con 0x7f61bc09db00 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:13.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 4 set_mon_vals no callback set 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 1 -- 172.21.15.5:0/2631467129 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b800dd60 con 0x7f61bc09db00 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61b3fff700 10 monclient: dump: 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.393 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 10 monclient: shutdown 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- 172.21.15.5:0/2631467129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09db00 msgr2=0x7f61bc09eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 --2- 172.21.15.5:0/2631467129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09db00 0x7f61bc09eff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0a1a70 tx=0x7f61b800d330 comp rx=0 tx=0).stop 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- 172.21.15.5:0/2631467129 shutdown_connections 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 --2- 172.21.15.5:0/2631467129 >> v2:172.21.15.143:3300/0 conn(0x7f61bc137ab0 0x7f61bc139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 --2- 172.21.15.5:0/2631467129 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09db00 0x7f61bc09eff0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- 172.21.15.5:0/2631467129 >> 172.21.15.5:0/2631467129 conn(0x7f61bc0a45d0 msgr2=0x7f61bc0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- 172.21.15.5:0/2631467129 shutdown_connections 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.373+0000 7f61c21c8700 1 -- 172.21.15.5:0/2631467129 wait complete. 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: monmap: 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 librados: starting msgr at 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 librados: starting objecter 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 Processor -- start 2024-09-16T11:08:13.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- start start 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 librados: setting wanted keys 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 librados: calling monclient init 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: init 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-a con 0x7f61bc09d720 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 0x7f61bc11c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-c con 0x7f61bc09db00 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61bc137ab0 0x7f61bc11c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): picked mon.noname-b con 0x7f61bc137ab0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc11fa00 con 0x7f61bc09d720 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc09f6b0 con 0x7f61bc137ab0 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc09f160 con 0x7f61bc09db00 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient(hunting): authenticate will time out at 1951.957516s 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51894/0 (socket says 172.21.15.5:51894) 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 -- 172.21.15.5:0/3359623946 learned_addr learned my addr 172.21.15.5:0/3359623946 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): get_auth_request con 0x7f61bc09d720 auth_method 0 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 0x7f61bc11c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): get_auth_request con 0x7f61bc09db00 auth_method 0 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c19c7700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61bc137ab0 0x7f61bc11c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c19c7700 10 monclient(hunting): get_auth_request con 0x7f61bc137ab0 auth_method 0 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c19c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c19c7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c19c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient(hunting): handle_auth_done global_id 74733 payload 824 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 -- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 msgr2=0x7f61bc11c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 0x7f61bc11c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 -- 172.21.15.5:0/3359623946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61bc137ab0 msgr2=0x7f61bc11c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61bc137ab0 0x7f61bc11c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 -- 172.21.15.5:0/3359623946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4009a70 con 0x7f61bc09d720 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c09c5700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 0x7f61bc11c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c11c6700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c500 tx=0x7f61ac00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 client.?.objecter ms_handle_connect 0x7f61bc09d720 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0052d0 con 0x7f61bc09d720 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: dump: 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.400 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 5 monclient: authenticate success, global_id 74733 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac005430 con 0x7f61bc09d720 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _renew_subs 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b80085f0 con 0x7f61bc09d720 2024-09-16T11:08:13.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0055a0 con 0x7f61bc09d720 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 10 monclient: dump: 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.402 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b37fe700 4 set_mon_vals no callback set 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 client.74733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _renew_subs 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc11f720 con 0x7f61bc09d720 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61c21c8700 1 librados: init done 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.377+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f61ac0036a0 con 0x7f61bc09d720 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c21c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c21c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c21c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc11f160 con 0x7f61bc09d720 2024-09-16T11:08:13.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 1 --2- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 0x7f61a0039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f61ac046ef0 con 0x7f61bc09d720 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 10 client.74733.objecter ms_dispatch 0x7f61bc000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 3 client.74733.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 3 client.74733.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 20 client.74733.objecter dump_active .. 0 homeless 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c09c5700 1 --2- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 0x7f61a0039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c09c5700 10 monclient: get_auth_request con 0x7f61a0036f60 auth_method 0 2024-09-16T11:08:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61c09c5700 1 --2- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 0x7f61a0039420 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f61b8000c00 tx=0x7f61b800a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.381+0000 7f61b1ffb700 10 client.74733.objecter ms_handle_connect 0x7f61a0036f60 2024-09-16T11:08:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.385+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f61ac010b10 con 0x7f61bc09d720 2024-09-16T11:08:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.385+0000 7f61b1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.385+0000 7f61b1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:13.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.553+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f61bc09df30 con 0x7f61a0036f60 2024-09-16T11:08:13.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.553+0000 7f61b1ffb700 1 -- 172.21.15.5:0/3359623946 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61b800dcc0 con 0x7f61a0036f60 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 10 librados: watch_flush enter 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 10 librados: watch_flush exit 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 20 client.74733.objecter shutdown clearing up homeless session... 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 10 client.74733.objecter successfully canceled tick 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 msgr2=0x7f61a0039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 0x7f61a0039420 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f61b8000c00 tx=0x7f61b800a040 comp rx=0 tx=0).stop 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 10 monclient: shutdown 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 msgr2=0x7f61bc11bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.557+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c500 tx=0x7f61ac00c810 comp rx=0 tx=0).stop 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 shutdown_connections 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61bc137ab0 0x7f61bc11c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f61bc09db00 0x7f61bc11c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> v2:172.21.15.143:6812/1098205034 conn(0x7f61a0036f60 0x7f61a0039420 unknown :-1 s=CLOSED pgs=3920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 --2- 172.21.15.5:0/3359623946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61bc09d720 0x7f61bc11bd50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 >> 172.21.15.5:0/3359623946 conn(0x7f61bc0a45d0 msgr2=0x7f61bc130df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 shutdown_connections 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 -- 172.21.15.5:0/3359623946 wait complete. 2024-09-16T11:08:13.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.561+0000 7f61c21c8700 1 librados: shutdown 2024-09-16T11:08:13.569 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T11:08:13.581 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T11:08:13.581 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 10 monclient: get_monmap_and_config 2024-09-16T11:08:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 10 monclient: monmap: 2024-09-16T11:08:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.705 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:13.703505+0000 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:13.703505+0000 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:13.706 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 1 Processor -- start 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 1 -- start start 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.697+0000 7feb92f1b700 10 monclient: init 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb8c0a3970 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c13aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-b con 0x7feb8c0a2200 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feb8c0a2660 0x7feb8c13d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-c con 0x7feb8c0a2660 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c136420 con 0x7feb8c0a3970 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c0a35f0 con 0x7feb8c0a2660 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0a3d50 con 0x7feb8c0a2200 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient(hunting): authenticate will time out at 1952.280268s 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c13aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91f19700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7feb8c0a3970 0x7feb8c135d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34574/0 (socket says 172.21.15.5:34574) 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91f19700 1 -- 172.21.15.5:0/2665977985 learned_addr learned my addr 172.21.15.5:0/2665977985 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): get_auth_request con 0x7feb8c0a2200 auth_method 0 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 1 --2- 172.21.15.5:0/2665977985 >> v2:172.21.15.143:3300/0 conn(0x7feb8c0a2660 0x7feb8c13d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): get_auth_request con 0x7feb8c0a2660 auth_method 0 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb9271a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient(hunting): handle_auth_done global_id 114179 payload 274 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 -- 172.21.15.5:0/2665977985 >> v2:172.21.15.143:3300/0 conn(0x7feb8c0a2660 msgr2=0x7feb8c13d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 --2- 172.21.15.5:0/2665977985 >> v2:172.21.15.143:3300/0 conn(0x7feb8c0a2660 0x7feb8c13d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 -- 172.21.15.5:0/2665977985 >> v1:172.21.15.5:6789/0 conn(0x7feb8c0a3970 legacy=0x7feb8c135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 monclient: found mon.noname-b 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 -- 172.21.15.5:0/2665977985 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0a49a0 con 0x7feb8c0a2200 2024-09-16T11:08:13.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 1 -- 172.21.15.5:0/2665977985 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828412896 0 0) 0x7feb8c136420 con 0x7feb8c0a3970 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb91718700 1 --2- 172.21.15.5:0/2665977985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c13aaf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009850 tx=0x7feb7c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7035454fb5c99d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 1 -- 172.21.15.5:0/2665977985 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c011830 con 0x7feb8c0a2200 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: dump: 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.723 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.724 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 5 monclient: authenticate success, global_id 114179 2024-09-16T11:08:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:13.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 1 -- 172.21.15.5:0/2665977985 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c011990 con 0x7feb8c0a2200 2024-09-16T11:08:13.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:13.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 4 set_mon_vals no callback set 2024-09-16T11:08:13.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 1 -- 172.21.15.5:0/2665977985 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c011d40 con 0x7feb8c0a2200 2024-09-16T11:08:13.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:13.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb90f17700 10 monclient: dump: 2024-09-16T11:08:13.727 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.727 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.728 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: shutdown 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- 172.21.15.5:0/2665977985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 msgr2=0x7feb8c13aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 --2- 172.21.15.5:0/2665977985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c13aaf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009850 tx=0x7feb7c009b60 comp rx=0 tx=0).stop 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- 172.21.15.5:0/2665977985 shutdown_connections 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 --2- 172.21.15.5:0/2665977985 >> v2:172.21.15.143:3300/0 conn(0x7feb8c0a2660 0x7feb8c13d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 --2- 172.21.15.5:0/2665977985 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c13aaf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- 172.21.15.5:0/2665977985 >> 172.21.15.5:0/2665977985 conn(0x7feb8c0a8440 msgr2=0x7feb8c0a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- 172.21.15.5:0/2665977985 shutdown_connections 2024-09-16T11:08:13.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 -- 172.21.15.5:0/2665977985 wait complete. 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 10 monclient: monmap: 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:13.731 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:13.732 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 librados: starting msgr at 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 librados: starting objecter 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.701+0000 7feb92f1b700 1 Processor -- start 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- start start 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 librados: setting wanted keys 2024-09-16T11:08:13.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 librados: calling monclient init 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: init 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c12ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-c con 0x7feb8c0a2200 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 0x7feb8c12f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-a con 0x7feb8c0a2660 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb8c0a3970 0x7feb8c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): picked mon.noname-b con 0x7feb8c0a3970 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c136d80 con 0x7feb8c0a2660 2024-09-16T11:08:13.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c070c20 con 0x7feb8c0a3970 2024-09-16T11:08:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0a1f20 con 0x7feb8c0a2200 2024-09-16T11:08:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 0x7feb8c12f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient(hunting): authenticate will time out at 1952.285241s 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 0x7feb8c12f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51908/0 (socket says 172.21.15.5:51908) 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 -- 172.21.15.5:0/640742922 learned_addr learned my addr 172.21.15.5:0/640742922 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): get_auth_request con 0x7feb8c0a2660 auth_method 0 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c12ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): get_auth_request con 0x7feb8c0a2200 auth_method 0 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb8c0a3970 0x7feb8c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91f19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 10 monclient(hunting): get_auth_request con 0x7feb8c0a3970 auth_method 0 2024-09-16T11:08:13.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient(hunting): handle_auth_done global_id 74739 payload 824 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 -- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 msgr2=0x7feb8c12ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7feb8c0a2200 0x7feb8c12ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 -- 172.21.15.5:0/640742922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb8c0a3970 msgr2=0x7feb8c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb8c0a3970 0x7feb8c06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 -- 172.21.15.5:0/640742922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0094b0 con 0x7feb8c0a2660 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb9271a700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb8c0a3970 0x7feb8c06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 client.?.objecter ms_handle_connect 0x7feb8c0a2660 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb91718700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 0x7feb8c12f340 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0090b0 tx=0x7feb7c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c01aa10 con 0x7feb8c0a2660 2024-09-16T11:08:13.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: dump: 2024-09-16T11:08:13.743 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.743 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.743 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.744 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.745 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c01f070 con 0x7feb8c0a2660 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 5 monclient: authenticate success, global_id 74739 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _renew_subs 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c09f630 con 0x7feb8c0a2660 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c01adb0 con 0x7feb8c0a2660 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb827fc700 10 monclient: dump: 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.748 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.748 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.748 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.748 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.748 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 client.74739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _renew_subs 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb83fff700 4 set_mon_vals no callback set 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c0717f0 con 0x7feb8c0a2660 2024-09-16T11:08:13.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.705+0000 7feb92f1b700 1 librados: init done 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 10 client.74739.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 15 client.74739.objecter _session_command_op_assign -1 1 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 20 client.74739.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 10 client.74739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb8c12fe60 con 0x7feb8c0a2660 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7feb7c01b430 con 0x7feb8c0a2660 2024-09-16T11:08:13.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6812/1098205034 conn(0x7feb70036f60 0x7feb70039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7feb7c048b50 con 0x7feb8c0a2660 2024-09-16T11:08:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 client.74739.objecter ms_dispatch 0x7feb8c000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 3 client.74739.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 3 client.74739.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 client.74739.objecter checking command 1 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6800/3342511476 conn(0x7feb7003e060 0x7feb70040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter _get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 15 client.74739.objecter _session_command_op_remove -1 1 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter _get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 15 client.74739.objecter _session_command_op_assign 3 1 2024-09-16T11:08:13.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 4 2024-09-16T11:08:13.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 client.74739.objecter _send_command 1 2024-09-16T11:08:13.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 --> v2:172.21.15.143:6800/3342511476 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb70040b90 con 0x7feb7003e060 2024-09-16T11:08:13.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 20 client.74739.objecter dump_active .. 0 homeless 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb7c033060 con 0x7feb8c0a2660 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb91f19700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6812/1098205034 conn(0x7feb70036f60 0x7feb70039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb91f19700 10 monclient: get_auth_request con 0x7feb70036f60 auth_method 0 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb9271a700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6800/3342511476 conn(0x7feb7003e060 0x7feb70040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb9271a700 10 monclient: get_auth_request con 0x7feb7003e060 auth_method 0 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 client.74739.objecter ms_handle_connect 0x7feb70036f60 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb91f19700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6812/1098205034 conn(0x7feb70036f60 0x7feb70039420 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7feb88000f60 tx=0x7feb88007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb827fc700 10 client.74739.objecter ms_handle_connect 0x7feb7003e060 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.709+0000 7feb9271a700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6800/3342511476 conn(0x7feb7003e060 0x7feb70040500 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.713+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== osd.3 v2:172.21.15.143:6800/3342511476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7feb70040b90 con 0x7feb7003e060 2024-09-16T11:08:13.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.713+0000 7feb827fc700 10 client.74739.objecter ms_dispatch 0x7feb8c000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:13.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.713+0000 7feb827fc700 10 client.74739.objecter _finish_command 1 = system:0 2024-09-16T11:08:13.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.713+0000 7feb827fc700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 4 2024-09-16T11:08:13.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.713+0000 7feb827fc700 15 client.74739.objecter _session_command_op_remove 3 1 2024-09-16T11:08:13.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 client.74739.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:13.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 15 client.74739.objecter _session_command_op_assign -1 2 2024-09-16T11:08:13.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter _get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter _get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 15 client.74739.objecter _session_command_op_remove -1 2 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter get_session s=0x7feb7003d9b0 osd=3 3 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 15 client.74739.objecter _session_command_op_assign 3 2 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 4 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 client.74739.objecter _send_command 2 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 --> v2:172.21.15.143:6800/3342511476 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feb8c0a4070 con 0x7feb7003e060 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb827fc700 1 -- 172.21.15.5:0/640742922 <== osd.3 v2:172.21.15.143:6800/3342511476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7feb8400f040 con 0x7feb7003e060 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb827fc700 10 client.74739.objecter ms_dispatch 0x7feb8c000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb827fc700 10 client.74739.objecter _finish_command 2 = system:0 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb827fc700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 4 2024-09-16T11:08:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb827fc700 15 client.74739.objecter _session_command_op_remove 3 2 2024-09-16T11:08:13.759 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:13.759 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:13.759 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 librados: watch_flush enter 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 librados: watch_flush exit 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 client.74739.objecter close_session for osd.3 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6800/3342511476 conn(0x7feb7003e060 msgr2=0x7feb70040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6800/3342511476 conn(0x7feb7003e060 0x7feb70040500 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter put_session s=0x7feb7003d9b0 osd=3 1 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 client.74739.objecter shutdown clearing up homeless session... 2024-09-16T11:08:13.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 client.74739.objecter successfully canceled tick 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6812/1098205034 conn(0x7feb70036f60 msgr2=0x7feb70039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 --2- 172.21.15.5:0/640742922 >> v2:172.21.15.143:6812/1098205034 conn(0x7feb70036f60 0x7feb70039420 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7feb88000f60 tx=0x7feb88007e00 comp rx=0 tx=0).stop 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 10 monclient: shutdown 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 msgr2=0x7feb8c12f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 --2- 172.21.15.5:0/640742922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feb8c0a2660 0x7feb8c12f340 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0090b0 tx=0x7feb7c004770 comp rx=0 tx=0).stop 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb9271a700 1 -- 172.21.15.5:0/640742922 reap_dead start 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 shutdown_connections 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 >> 172.21.15.5:0/640742922 conn(0x7feb8c0a8440 msgr2=0x7feb8c0a0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 shutdown_connections 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 -- 172.21.15.5:0/640742922 wait complete. 2024-09-16T11:08:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.733+0000 7feb92f1b700 1 librados: shutdown 2024-09-16T11:08:13.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T11:08:13.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:13.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 10 monclient: monmap: 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:13.879955+0000 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:13.879955+0000 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:13.882 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:13.883 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:13.883 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:13.883 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:13.883 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 1 Processor -- start 2024-09-16T11:08:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 1 -- start start 2024-09-16T11:08:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.873+0000 7f0853fff700 10 monclient: init 2024-09-16T11:08:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 0x7f084c0234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-c con 0x7f084c0b63f0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f084c0b67d0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f084c0b6c20 0x7f084c0b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-a con 0x7f084c0b6c20 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0b6030 con 0x7f084c0b67d0 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c020000 con 0x7f084c0b6c20 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c01fce0 con 0x7f084c0b63f0 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient(hunting): authenticate will time out at 1952.456367s 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f084c0b6c20 0x7f084c0b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f084c0b6c20 0x7f084c0b9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51920/0 (socket says 172.21.15.5:51920) 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 -- 172.21.15.5:0/1984623887 learned_addr learned my addr 172.21.15.5:0/1984623887 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): get_auth_request con 0x7f084c0b6c20 auth_method 0 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 1 --2- 172.21.15.5:0/1984623887 >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 0x7f084c0234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 10 monclient(hunting): get_auth_request con 0x7f084c0b63f0 auth_method 0 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient(hunting): handle_auth_done global_id 74742 payload 274 2024-09-16T11:08:13.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 -- 172.21.15.5:0/1984623887 >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 msgr2=0x7f084c0234f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 --2- 172.21.15.5:0/1984623887 >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 0x7f084c0234f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 -- 172.21.15.5:0/1984623887 >> v1:172.21.15.143:6789/0 conn(0x7f084c0b67d0 legacy=0x7f084c0b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 -- 172.21.15.5:0/1984623887 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0222e0 con 0x7f084c0b6c20 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0852ffd700 1 --2- 172.21.15.5:0/1984623887 >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 0x7f084c0234f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f08537fe700 1 --2- 172.21.15.5:0/1984623887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b6c20 0x7f084c0b9c40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f085404f940 tx=0x7f0854066b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a0340c5af4ed24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 1 -- 172.21.15.5:0/1984623887 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854066e00 con 0x7f084c0b6c20 2024-09-16T11:08:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: dump: 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 5 monclient: authenticate success, global_id 74742 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 1 -- 172.21.15.5:0/1984623887 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08540008d0 con 0x7f084c0b6c20 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 4 set_mon_vals no callback set 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 1 -- 172.21.15.5:0/1984623887 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854067310 con 0x7f084c0b6c20 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0851ffb700 10 monclient: dump: 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 10 monclient: shutdown 2024-09-16T11:08:13.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- 172.21.15.5:0/1984623887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b6c20 msgr2=0x7f084c0b9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 --2- 172.21.15.5:0/1984623887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b6c20 0x7f084c0b9c40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f085404f940 tx=0x7f0854066b60 comp rx=0 tx=0).stop 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- 172.21.15.5:0/1984623887 shutdown_connections 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 --2- 172.21.15.5:0/1984623887 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b6c20 0x7f084c0b9c40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 --2- 172.21.15.5:0/1984623887 >> v2:172.21.15.143:3300/0 conn(0x7f084c0b63f0 0x7f084c0234f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- 172.21.15.5:0/1984623887 >> 172.21.15.5:0/1984623887 conn(0x7f084c0064b0 msgr2=0x7f084c0068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- 172.21.15.5:0/1984623887 shutdown_connections 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.877+0000 7f0853fff700 1 -- 172.21.15.5:0/1984623887 wait complete. 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient: build_initial_monmap 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient: monmap: 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:13.896 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 librados: starting msgr at 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 librados: starting objecter 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 Processor -- start 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 -- start start 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 librados: setting wanted keys 2024-09-16T11:08:13.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 librados: calling monclient init 2024-09-16T11:08:13.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient: init 2024-09-16T11:08:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T11:08:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-a con 0x7f084c0b63f0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 0x7f084c0a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-b con 0x7f084c0b67d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f084c0b6c20 0x7f084c0a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): picked mon.noname-c con 0x7f084c0b6c20 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0b9d90 con 0x7f084c0b63f0 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0a6590 con 0x7f084c0b67d0 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0a50e0 con 0x7f084c0b6c20 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0853fff700 10 monclient(hunting): authenticate will time out at 1952.462977s 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51930/0 (socket says 172.21.15.5:51930) 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 -- 172.21.15.5:0/3486591142 learned_addr learned my addr 172.21.15.5:0/3486591142 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): get_auth_request con 0x7f084c0b63f0 auth_method 0 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f084c0b6c20 0x7f084c0a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): get_auth_request con 0x7f084c0b6c20 auth_method 0 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 0x7f084c0a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 10 monclient(hunting): get_auth_request con 0x7f084c0b67d0 auth_method 0 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient(hunting): handle_auth_done global_id 74745 payload 824 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 -- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f084c0b6c20 msgr2=0x7f084c0a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f084c0b6c20 0x7f084c0a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 -- 172.21.15.5:0/3486591142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 msgr2=0x7f084c0a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 0x7f084c0a28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient: _finish_hunting 0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 monclient: found mon.noname-a 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 -- 172.21.15.5:0/3486591142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0854069b60 con 0x7f084c0b63f0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f08527fc700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 0x7f084c0a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f0852ffd700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f084800e880 tx=0x7f084800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f083bfff700 10 client.?.objecter ms_handle_connect 0x7f084c0b63f0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.881+0000 7f083bfff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848013610 con 0x7f084c0b63f0 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:13.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: dump: 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: _finish_auth 0 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:13.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0848013770 con 0x7f084c0b63f0 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 5 monclient: authenticate success, global_id 74745 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848013b00 con 0x7f084c0b63f0 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _renew_subs 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f08517fa700 4 set_mon_vals no callback set 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c020870 con 0x7f084c0b63f0 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 monclient: dump: 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 client.74745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _renew_subs 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c0a62b0 con 0x7f084c0b63f0 2024-09-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 1 librados: init done 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f0848013c60 con 0x7f084c0b63f0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 --2- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 0x7f0840039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f08480478f0 con 0x7f084c0b63f0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 client.74745.objecter ms_dispatch 0x7f0854000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 3 client.74745.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 3 client.74745.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 20 client.74745.objecter dump_active .. 0 homeless 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c0a6930 con 0x7f084c0b63f0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f08527fc700 1 --2- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 0x7f0840039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f08527fc700 10 monclient: get_auth_request con 0x7f0840036f60 auth_method 0 2024-09-16T11:08:13.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f083bfff700 10 client.74745.objecter ms_handle_connect 0x7f0840036f60 2024-09-16T11:08:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.885+0000 7f08527fc700 1 --2- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 0x7f0840039420 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 crypto rx=0x7f084c0a4110 tx=0x7f0844009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.889+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f084801b9a0 con 0x7f084c0b63f0 2024-09-16T11:08:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.893+0000 7f083bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:13.893+0000 7f083bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.057+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f084c0ba180 con 0x7f0840036f60 2024-09-16T11:08:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f083bfff700 1 -- 172.21.15.5:0/3486591142 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0844001c60 con 0x7f0840036f60 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 10 librados: watch_flush enter 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 10 librados: watch_flush exit 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 20 client.74745.objecter shutdown clearing up homeless session... 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 10 client.74745.objecter successfully canceled tick 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 msgr2=0x7f0840039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 0x7f0840039420 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 crypto rx=0x7f084c0a4110 tx=0x7f0844009040 comp rx=0 tx=0).stop 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 10 monclient: shutdown 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 msgr2=0x7f084c14e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f084800e880 tx=0x7f084800eb90 comp rx=0 tx=0).stop 2024-09-16T11:08:14.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 shutdown_connections 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f084c0b6c20 0x7f084c0a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c0b67d0 0x7f084c0a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> v2:172.21.15.143:6812/1098205034 conn(0x7f0840036f60 0x7f0840039420 unknown :-1 s=CLOSED pgs=3922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 --2- 172.21.15.5:0/3486591142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f084c0b63f0 0x7f084c14e8d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 >> 172.21.15.5:0/3486591142 conn(0x7f084c0064b0 msgr2=0x7f084c0b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 shutdown_connections 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 -- 172.21.15.5:0/3486591142 wait complete. 2024-09-16T11:08:14.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.061+0000 7f0853fff700 1 librados: shutdown 2024-09-16T11:08:14.072 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T11:08:14.083 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T11:08:14.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:14.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: monmap: 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:14.199086+0000 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:14.199086+0000 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:14.202 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 1 Processor -- start 2024-09-16T11:08:14.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 1 -- start start 2024-09-16T11:08:14.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: init 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.193+0000 7ffa4b59e700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa440067a0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa44024a00 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa44007290 0x7ffa44024400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ffa44007290 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa44023360 con 0x7ffa44024a00 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa440235b0 con 0x7ffa440067a0 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa44023890 con 0x7ffa44007290 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient(hunting): authenticate will time out at 1952.776267s 2024-09-16T11:08:14.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa44007290 0x7ffa44024400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa49d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ffa44024a00 0x7ffa440c0d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34584/0 (socket says 172.21.15.5:34584) 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa49d9b700 1 -- 172.21.15.5:0/67643453 learned_addr learned my addr 172.21.15.5:0/67643453 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): get_auth_request con 0x7ffa44007290 auth_method 0 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_done global_id 74748 payload 274 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/67643453 >> v1:172.21.15.143:6789/0 conn(0x7ffa440067a0 legacy=0x7ffa44006b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/67643453 >> v1:172.21.15.5:6789/0 conn(0x7ffa44024a00 legacy=0x7ffa440c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 monclient: found mon.noname-a 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/67643453 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa44023b70 con 0x7ffa44007290 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 1 -- 172.21.15.5:0/67643453 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918490413 0 0) 0x7ffa44023360 con 0x7ffa44024a00 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4ad9d700 1 --2- 172.21.15.5:0/67643453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44007290 0x7ffa44024400 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c04ffd0 tx=0x7ffa4c066b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b42cea3d10e677e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 1 -- 172.21.15.5:0/67643453 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0008d0 con 0x7ffa44007290 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:14.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: dump: 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 5 monclient: authenticate success, global_id 74748 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 1 -- 172.21.15.5:0/67643453 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c067310 con 0x7ffa44007290 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 1 -- 172.21.15.5:0/67643453 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c000a30 con 0x7ffa44007290 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4959a700 10 monclient: dump: 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.211 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 4 set_mon_vals no callback set 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 10 monclient: shutdown 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- 172.21.15.5:0/67643453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44007290 msgr2=0x7ffa44024400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 --2- 172.21.15.5:0/67643453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44007290 0x7ffa44024400 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c04ffd0 tx=0x7ffa4c066b60 comp rx=0 tx=0).stop 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- 172.21.15.5:0/67643453 shutdown_connections 2024-09-16T11:08:14.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 --2- 172.21.15.5:0/67643453 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44007290 0x7ffa44024400 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- 172.21.15.5:0/67643453 >> 172.21.15.5:0/67643453 conn(0x7ffa440274b0 msgr2=0x7ffa440278c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- 172.21.15.5:0/67643453 shutdown_connections 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.197+0000 7ffa4b59e700 1 -- 172.21.15.5:0/67643453 wait complete. 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: monmap: 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:14.213 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:14.214 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 librados: starting msgr at 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 librados: starting objecter 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 Processor -- start 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- start start 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 librados: setting wanted keys 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 librados: calling monclient init 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: init 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa440067a0 0x7ffa44150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-b con 0x7ffa440067a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa44007290 0x7ffa44151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-c con 0x7ffa44007290 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 0x7ffa440bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ffa44024a00 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440a5420 con 0x7ffa44024a00 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa440c2150 con 0x7ffa440067a0 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4401e690 con 0x7ffa44007290 2024-09-16T11:08:14.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient(hunting): authenticate will time out at 1952.781833s 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 0x7ffa440bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 0x7ffa440bfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51942/0 (socket says 172.21.15.5:51942) 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/2883736114 learned_addr learned my addr 172.21.15.5:0/2883736114 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): get_auth_request con 0x7ffa44024a00 auth_method 0 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa44007290 0x7ffa44151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa440067a0 0x7ffa44150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): get_auth_request con 0x7ffa440067a0 auth_method 0 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): get_auth_request con 0x7ffa44007290 auth_method 0 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa49d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient(hunting): handle_auth_done global_id 74754 payload 824 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa44007290 msgr2=0x7ffa44151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7ffa44007290 0x7ffa44151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/2883736114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa440067a0 msgr2=0x7ffa44150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa440067a0 0x7ffa44150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 monclient: found mon.noname-a 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/2883736114 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c069b60 con 0x7ffa44024a00 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4ad9d700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 0x7ffa440bfc00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0731d0 tx=0x7ffa4c0690c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 client.?.objecter ms_handle_connect 0x7ffa44024a00 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c066090 con 0x7ffa44024a00 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: dump: 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.226 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4c0661f0 con 0x7ffa44024a00 2024-09-16T11:08:14.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 5 monclient: authenticate success, global_id 74754 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _renew_subs 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa44021980 con 0x7ffa44024a00 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c066560 con 0x7ffa44024a00 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa3affd700 10 monclient: dump: 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.228 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa48d99700 4 set_mon_vals no callback set 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 client.74754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _renew_subs 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa440c3bc0 con 0x7ffa44024a00 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.201+0000 7ffa4b59e700 1 librados: init done 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7ffa4c0721d0 con 0x7ffa44024a00 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 10 client.74754.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 15 client.74754.objecter _session_command_op_assign -1 1 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 20 client.74754.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 10 client.74754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa440c3bc0 con 0x7ffa44024a00 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa2c036f60 0x7ffa2c039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7ffa4c0c4050 con 0x7ffa44024a00 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 client.74754.objecter ms_dispatch 0x7ffa4c000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 3 client.74754.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 3 client.74754.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 client.74754.objecter checking command 1 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6808/3348750750 conn(0x7ffa2c03e060 0x7ffa2c040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter _get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 15 client.74754.objecter _session_command_op_remove -1 1 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter _get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 15 client.74754.objecter _session_command_op_assign 4 1 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 4 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 client.74754.objecter _send_command 1 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 --> v2:172.21.15.143:6808/3348750750 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2c040b90 con 0x7ffa2c03e060 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 20 client.74754.objecter dump_active .. 0 homeless 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa4c071450 con 0x7ffa44024a00 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4a59c700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa2c036f60 0x7ffa2c039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4a59c700 10 monclient: get_auth_request con 0x7ffa2c036f60 auth_method 0 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa49d9b700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6808/3348750750 conn(0x7ffa2c03e060 0x7ffa2c040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa49d9b700 10 monclient: get_auth_request con 0x7ffa2c03e060 auth_method 0 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa4a59c700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa2c036f60 0x7ffa2c039420 secure :-1 s=READY pgs=3923 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c000f60 tx=0x7ffa3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 client.74754.objecter ms_handle_connect 0x7ffa2c036f60 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa49d9b700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6808/3348750750 conn(0x7ffa2c03e060 0x7ffa2c040500 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.205+0000 7ffa3affd700 10 client.74754.objecter ms_handle_connect 0x7ffa2c03e060 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.209+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== osd.4 v2:172.21.15.143:6808/3348750750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ffa2c040b90 con 0x7ffa2c03e060 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.209+0000 7ffa3affd700 10 client.74754.objecter ms_dispatch 0x7ffa4c000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.209+0000 7ffa3affd700 10 client.74754.objecter _finish_command 1 = system:0 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.209+0000 7ffa3affd700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 4 2024-09-16T11:08:14.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.209+0000 7ffa3affd700 15 client.74754.objecter _session_command_op_remove 4 1 2024-09-16T11:08:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 client.74754.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 15 client.74754.objecter _session_command_op_assign -1 2 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter _get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter _get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 15 client.74754.objecter _session_command_op_remove -1 2 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter get_session s=0x7ffa2c03d9b0 osd=4 3 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 15 client.74754.objecter _session_command_op_assign 4 2 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 4 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 client.74754.objecter _send_command 2 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 --> v2:172.21.15.143:6808/3348750750 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ffa44024400 con 0x7ffa2c03e060 2024-09-16T11:08:14.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa3affd700 1 -- 172.21.15.5:0/2883736114 <== osd.4 v2:172.21.15.143:6808/3348750750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ffa44024400 con 0x7ffa2c03e060 2024-09-16T11:08:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa3affd700 10 client.74754.objecter ms_dispatch 0x7ffa4c000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa3affd700 10 client.74754.objecter _finish_command 2 = system:0 2024-09-16T11:08:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa3affd700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 4 2024-09-16T11:08:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa3affd700 15 client.74754.objecter _session_command_op_remove 4 2 2024-09-16T11:08:14.237 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:14.238 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:14.238 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 librados: watch_flush enter 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 librados: watch_flush exit 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 client.74754.objecter close_session for osd.4 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6808/3348750750 conn(0x7ffa2c03e060 msgr2=0x7ffa2c040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6808/3348750750 conn(0x7ffa2c03e060 0x7ffa2c040500 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter put_session s=0x7ffa2c03d9b0 osd=4 1 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 client.74754.objecter shutdown clearing up homeless session... 2024-09-16T11:08:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 client.74754.objecter successfully canceled tick 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa2c036f60 msgr2=0x7ffa2c039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 --2- 172.21.15.5:0/2883736114 >> v2:172.21.15.143:6812/1098205034 conn(0x7ffa2c036f60 0x7ffa2c039420 secure :-1 s=READY pgs=3923 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c000f60 tx=0x7ffa3c009040 comp rx=0 tx=0).stop 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 10 monclient: shutdown 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 msgr2=0x7ffa440bfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 --2- 172.21.15.5:0/2883736114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa44024a00 0x7ffa440bfc00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0731d0 tx=0x7ffa4c0690c0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4ad9d700 1 -- 172.21.15.5:0/2883736114 reap_dead start 2024-09-16T11:08:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 shutdown_connections 2024-09-16T11:08:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 >> 172.21.15.5:0/2883736114 conn(0x7ffa440274b0 msgr2=0x7ffa44020720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 shutdown_connections 2024-09-16T11:08:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 -- 172.21.15.5:0/2883736114 wait complete. 2024-09-16T11:08:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.225+0000 7ffa4b59e700 1 librados: shutdown 2024-09-16T11:08:14.245 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T11:08:14.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: get_monmap_and_config 2024-09-16T11:08:14.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: monmap: 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:14.355111+0000 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:14.355111+0000 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:14.357 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 Processor -- start 2024-09-16T11:08:14.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 -- start start 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: init 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f165c09e8e0 0x7f165c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-a con 0x7f165c09e8e0 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f165c09d6a0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 0x7f165c09f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-c con 0x7f165c09dfd0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c070e40 con 0x7f165c09d6a0 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c0a01d0 con 0x7f165c09e8e0 2024-09-16T11:08:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c09f740 con 0x7f165c09dfd0 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1662fa3700 10 monclient(hunting): authenticate will time out at 1952.931449s 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f165c09e8e0 0x7f165c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f165c09e8e0 0x7f165c09ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51958/0 (socket says 172.21.15.5:51958) 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 1 -- 172.21.15.5:0/242390579 learned_addr learned my addr 172.21.15.5:0/242390579 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 10 monclient(hunting): get_auth_request con 0x7f165c09e8e0 auth_method 0 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f1661fa1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f16627a2700 1 --2- 172.21.15.5:0/242390579 >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 0x7f165c09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f16627a2700 10 monclient(hunting): get_auth_request con 0x7f165c09dfd0 auth_method 0 2024-09-16T11:08:14.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f16627a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f16627a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.349+0000 7f16627a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient(hunting): handle_auth_done global_id 74757 payload 274 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 -- 172.21.15.5:0/242390579 >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 msgr2=0x7f165c09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 --2- 172.21.15.5:0/242390579 >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 0x7f165c09f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 -- 172.21.15.5:0/242390579 >> v1:172.21.15.143:6789/0 conn(0x7f165c09d6a0 legacy=0x7f165c1336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 monclient: found mon.noname-a 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 -- 172.21.15.5:0/242390579 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c09fef0 con 0x7f165c09e8e0 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f16627a2700 1 --2- 172.21.15.5:0/242390579 >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 0x7f165c09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1661fa1700 1 --2- 172.21.15.5:0/242390579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09e8e0 0x7f165c09ecc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1650009910 tx=0x7f1650009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922ed886ba5a0d84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 1 -- 172.21.15.5:0/242390579 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650011830 con 0x7f165c09e8e0 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: dump: 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.365 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 5 monclient: authenticate success, global_id 74757 2024-09-16T11:08:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 1 -- 172.21.15.5:0/242390579 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1650011990 con 0x7f165c09e8e0 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 4 set_mon_vals no callback set 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 1 -- 172.21.15.5:0/242390579 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650011d40 con 0x7f165c09e8e0 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1660f9f700 10 monclient: dump: 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 10 monclient: shutdown 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- 172.21.15.5:0/242390579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09e8e0 msgr2=0x7f165c09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 --2- 172.21.15.5:0/242390579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09e8e0 0x7f165c09ecc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1650009910 tx=0x7f1650009c20 comp rx=0 tx=0).stop 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- 172.21.15.5:0/242390579 shutdown_connections 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 --2- 172.21.15.5:0/242390579 >> v2:172.21.15.143:3300/0 conn(0x7f165c09dfd0 0x7f165c09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 --2- 172.21.15.5:0/242390579 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09e8e0 0x7f165c09ecc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- 172.21.15.5:0/242390579 >> 172.21.15.5:0/242390579 conn(0x7f165c0a45d0 msgr2=0x7f165c0a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- 172.21.15.5:0/242390579 shutdown_connections 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- 172.21.15.5:0/242390579 wait complete. 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 10 monclient: monmap: 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 librados: starting msgr at 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 librados: starting objecter 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 Processor -- start 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 -- start start 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 librados: setting wanted keys 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 1 librados: calling monclient init 2024-09-16T11:08:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.353+0000 7f1662fa3700 10 monclient: init 2024-09-16T11:08:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-a con 0x7f165c09d6a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 0x7f165c132a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-c con 0x7f165c09dfd0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f165c09e8e0 0x7f165c132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): picked mon.noname-b con 0x7f165c09e8e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c1c8780 con 0x7f165c09d6a0 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c133750 con 0x7f165c09e8e0 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0702c0 con 0x7f165c09dfd0 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient(hunting): authenticate will time out at 1952.936190s 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 0x7f165c132a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51970/0 (socket says 172.21.15.5:51970) 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 1 -- 172.21.15.5:0/2906964920 learned_addr learned my addr 172.21.15.5:0/2906964920 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): get_auth_request con 0x7f165c09d6a0 auth_method 0 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): get_auth_request con 0x7f165c09dfd0 auth_method 0 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f165c09e8e0 0x7f165c132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): get_auth_request con 0x7f165c09e8e0 auth_method 0 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16627a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient(hunting): handle_auth_done global_id 114191 payload 824 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 -- 172.21.15.5:0/2906964920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f165c09e8e0 msgr2=0x7f165c132f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f165c09e8e0 0x7f165c132f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 -- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 msgr2=0x7f165c1c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 monclient: found mon.noname-c 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 -- 172.21.15.5:0/2906964920 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654009a70 con 0x7f165c09dfd0 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1661fa1700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:14.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f16617a0700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 0x7f165c132a10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f165800c400 tx=0x7f165800c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 client.?.objecter ms_handle_connect 0x7f165c09dfd0 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580050d0 con 0x7f165c09dfd0 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: dump: 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.379 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1658005250 con 0x7f165c09dfd0 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 5 monclient: authenticate success, global_id 114191 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _renew_subs 2024-09-16T11:08:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 -- 172.21.15.5:0/2906964920 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1650009570 con 0x7f165c09dfd0 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658006880 con 0x7f165c09dfd0 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 monclient: dump: 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.381 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164ffff700 4 set_mon_vals no callback set 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 client.114191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _renew_subs 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 -- 172.21.15.5:0/2906964920 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c1200f0 con 0x7f165c09dfd0 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f1662fa3700 1 librados: init done 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f16580036a0 con 0x7f165c09dfd0 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 --2- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 0x7f1640039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f1658046bd0 con 0x7f165c09dfd0 2024-09-16T11:08:14.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 10 client.114191.objecter ms_dispatch 0x7f165c000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 3 client.114191.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f164e7fc700 3 client.114191.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f163ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f163ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f163ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.357+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1638005100 con 0x7f165c09dfd0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.361+0000 7f164e7fc700 20 client.114191.objecter dump_active .. 0 homeless 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.361+0000 7f1661fa1700 1 --2- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 0x7f1640039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.361+0000 7f1661fa1700 10 monclient: get_auth_request con 0x7f1640036f60 auth_method 0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.361+0000 7f1661fa1700 1 --2- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 0x7f1640039420 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 crypto rx=0x7f1650009150 tx=0x7f1650008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.361+0000 7f164e7fc700 10 client.114191.objecter ms_handle_connect 0x7f1640036f60 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.365+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f16580108d0 con 0x7f165c09dfd0 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.365+0000 7f164e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.365+0000 7f164e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:14.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.533+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1638000bc0 con 0x7f1640036f60 2024-09-16T11:08:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f164e7fc700 1 -- 172.21.15.5:0/2906964920 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1638000bc0 con 0x7f1640036f60 2024-09-16T11:08:14.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 10 librados: watch_flush enter 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 10 librados: watch_flush exit 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 20 client.114191.objecter shutdown clearing up homeless session... 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 10 client.114191.objecter successfully canceled tick 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 msgr2=0x7f1640039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 0x7f1640039420 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 crypto rx=0x7f1650009150 tx=0x7f1650008030 comp rx=0 tx=0).stop 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 10 monclient: shutdown 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 msgr2=0x7f165c132a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 0x7f165c132a10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f165800c400 tx=0x7f165800c710 comp rx=0 tx=0).stop 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 shutdown_connections 2024-09-16T11:08:14.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f165c09e8e0 0x7f165c132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f165c09dfd0 0x7f165c132a10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1640036f60 0x7f1640039420 unknown :-1 s=CLOSED pgs=3924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 --2- 172.21.15.5:0/2906964920 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f165c09d6a0 0x7f165c1c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 >> 172.21.15.5:0/2906964920 conn(0x7f165c0a45d0 msgr2=0x7f165c0a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 shutdown_connections 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 -- 172.21.15.5:0/2906964920 wait complete. 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.537+0000 7f163ffff700 1 librados: shutdown 2024-09-16T11:08:14.547 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T11:08:14.558 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T11:08:14.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 10 monclient: monmap: 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.681 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:14.679169+0000 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:14.679169+0000 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:14.682 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:14.683 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 1 Processor -- start 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 1 -- start start 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.673+0000 7f151ffff700 10 monclient: init 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15200a2cf0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15200a37e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:14.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f152013c620 0x7f152013ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f152013c620 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15200a3d70 con 0x7f15200a37e0 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15200a4220 con 0x7f15200a2cf0 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200a4c10 con 0x7f152013c620 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient(hunting): authenticate will time out at 1953.256656s 2024-09-16T11:08:14.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151e7fc700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f15200a37e0 0x7f152013af00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34598/0 (socket says 172.21.15.5:34598) 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151e7fc700 1 -- 172.21.15.5:0/2435873515 learned_addr learned my addr 172.21.15.5:0/2435873515 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 --2- 172.21.15.5:0/2435873515 >> v2:172.21.15.143:3300/0 conn(0x7f152013c620 0x7f152013ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): get_auth_request con 0x7f152013c620 auth_method 0 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 1 -- 172.21.15.5:0/2435873515 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519463868 0 0) 0x7f15200a3d70 con 0x7f15200a37e0 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient(hunting): _negotiate 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient(hunting): my global_id is 74763 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 1 -- 172.21.15.5:0/2435873515 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15040035d0 con 0x7f15200a37e0 2024-09-16T11:08:14.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient(hunting): handle_auth_done global_id 94441 payload 274 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 -- 172.21.15.5:0/2435873515 >> v1:172.21.15.143:6789/0 conn(0x7f15200a2cf0 legacy=0x7f15200a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 -- 172.21.15.5:0/2435873515 >> v1:172.21.15.5:6789/0 conn(0x7f15200a37e0 legacy=0x7f152013af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 monclient: found mon.noname-c 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 -- 172.21.15.5:0/2435873515 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152009ff40 con 0x7f152013c620 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151f7fe700 1 --2- 172.21.15.5:0/2435873515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f152013ea10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1518009d90 tx=0x7f151800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ed92fc48e6cc5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 1 -- 172.21.15.5:0/2435873515 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151800e070 con 0x7f152013c620 2024-09-16T11:08:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: dump: 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.703 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 5 monclient: authenticate success, global_id 94441 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 1 -- 172.21.15.5:0/2435873515 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151800bb20 con 0x7f152013c620 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 4 set_mon_vals no callback set 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 1 -- 172.21.15.5:0/2435873515 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151800be90 con 0x7f152013c620 2024-09-16T11:08:14.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151dffb700 10 monclient: dump: 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 10 monclient: shutdown 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- 172.21.15.5:0/2435873515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 msgr2=0x7f152013ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 --2- 172.21.15.5:0/2435873515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f152013ea10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1518009d90 tx=0x7f151800b330 comp rx=0 tx=0).stop 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- 172.21.15.5:0/2435873515 shutdown_connections 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 --2- 172.21.15.5:0/2435873515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f152013ea10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- 172.21.15.5:0/2435873515 >> 172.21.15.5:0/2435873515 conn(0x7f15200a4df0 msgr2=0x7f15200a5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- 172.21.15.5:0/2435873515 shutdown_connections 2024-09-16T11:08:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.677+0000 7f151ffff700 1 -- 172.21.15.5:0/2435873515 wait complete. 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient: monmap: 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.707 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 librados: starting msgr at 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 librados: starting objecter 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 Processor -- start 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 -- start start 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 librados: setting wanted keys 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 librados: calling monclient init 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient: init 2024-09-16T11:08:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15200a2cf0 0x7f15201ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f15200a2cf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 0x7f15201cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f15200a37e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f15201cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f152013c620 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:14.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1520120d80 con 0x7f15200a2cf0 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152009f4e0 con 0x7f152013c620 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152013c3d0 con 0x7f15200a37e0 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151ffff700 10 monclient(hunting): authenticate will time out at 1953.262849s 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15200a2cf0 0x7f15201ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 0x7f15201cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 0x7f15201cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:53050/0 (socket says 172.21.15.5:53050) 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 -- 172.21.15.5:0/1165330893 learned_addr learned my addr 172.21.15.5:0/1165330893 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): get_auth_request con 0x7f15200a37e0 auth_method 0 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15200a2cf0 0x7f15201ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51982/0 (socket says 172.21.15.5:51982) 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): get_auth_request con 0x7f15200a2cf0 auth_method 0 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151f7fe700 1 --2- 172.21.15.5:0/1165330893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f15201cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151f7fe700 10 monclient(hunting): get_auth_request con 0x7f152013c620 auth_method 0 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient(hunting): handle_auth_done global_id 114194 payload 824 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 -- 172.21.15.5:0/1165330893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 msgr2=0x7f15201cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 --2- 172.21.15.5:0/1165330893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f152013c620 0x7f15201cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 -- 172.21.15.5:0/1165330893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15200a2cf0 msgr2=0x7f15201ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 --2- 172.21.15.5:0/1165330893 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f15200a2cf0 0x7f15201ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 monclient: found mon.noname-c 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 -- 172.21.15.5:0/1165330893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1518009a70 con 0x7f15200a37e0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f151e7fc700 1 --2- 172.21.15.5:0/1165330893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 0x7f15201cace0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c700 tx=0x7f150c00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f15177fe700 10 client.?.objecter ms_handle_connect 0x7f15200a37e0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.681+0000 7f15177fe700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c005d90 con 0x7f15200a37e0 2024-09-16T11:08:14.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: dump: 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.715 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 5 monclient: authenticate success, global_id 114194 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c0070d0 con 0x7f15200a37e0 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 monclient: _renew_subs 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1520120aa0 con 0x7f15200a37e0 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c007480 con 0x7f15200a37e0 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:14.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 client.114194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: dump: 2024-09-16T11:08:14.717 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.718 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151cff9700 4 set_mon_vals no callback set 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 monclient: _renew_subs 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152013b5c0 con 0x7f15200a37e0 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 1 librados: init done 2024-09-16T11:08:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f150c0075e0 con 0x7f15200a37e0 2024-09-16T11:08:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 client.114194.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 15 client.114194.objecter _session_command_op_assign -1 1 2024-09-16T11:08:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 20 client.114194.objecter _get_session osd=-1 returning homeless 2024-09-16T11:08:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 client.114194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15200a44a0 con 0x7f15200a37e0 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1508036f60 0x7f1508039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f150c047000 con 0x7f15200a37e0 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 client.114194.objecter ms_dispatch 0x7f1520000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 3 client.114194.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 3 client.114194.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 client.114194.objecter checking command 1 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6804/3538568827 conn(0x7f150803e060 0x7f1508040500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter _get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 15 client.114194.objecter _session_command_op_remove -1 1 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter _get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 15 client.114194.objecter _session_command_op_assign 5 1 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 4 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 client.114194.objecter _send_command 1 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 --> v2:172.21.15.143:6804/3538568827 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1508040b90 con 0x7f150803e060 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 20 client.114194.objecter dump_active .. 0 homeless 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_get_version_reply(handle=1 version=694) v2 ==== 24+0+0 (secure 0 0 0) 0x7f150c049690 con 0x7f15200a37e0 2024-09-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 monclient: handle_get_version_reply finishing 1 version 694 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151effd700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1508036f60 0x7f1508039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151effd700 10 monclient: get_auth_request con 0x7f1508036f60 auth_method 0 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151f7fe700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6804/3538568827 conn(0x7f150803e060 0x7f1508040500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151f7fe700 10 monclient: get_auth_request con 0x7f150803e060 auth_method 0 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 client.114194.objecter ms_handle_connect 0x7f150803e060 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151f7fe700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6804/3538568827 conn(0x7f150803e060 0x7f1508040500 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f15177fe700 10 client.114194.objecter ms_handle_connect 0x7f1508036f60 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.685+0000 7f151effd700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1508036f60 0x7f1508039420 secure :-1 s=READY pgs=3926 cs=0 l=1 rev1=1 crypto rx=0x7f15100018a0 tx=0x7f1510009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.689+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== osd.5 v2:172.21.15.143:6804/3538568827 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f1508040b90 con 0x7f150803e060 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.689+0000 7f15177fe700 10 client.114194.objecter ms_dispatch 0x7f1520000fd0 command_reply(tid 1: 0 ) v1 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.689+0000 7f15177fe700 10 client.114194.objecter _finish_command 1 = system:0 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.689+0000 7f15177fe700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 4 2024-09-16T11:08:14.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.689+0000 7f15177fe700 15 client.114194.objecter _session_command_op_remove 5 1 2024-09-16T11:08:14.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 client.114194.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 15 client.114194.objecter _session_command_op_assign -1 2 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter _get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter _get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 15 client.114194.objecter _session_command_op_remove -1 2 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter get_session s=0x7f150803d9b0 osd=5 3 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 15 client.114194.objecter _session_command_op_assign 5 2 2024-09-16T11:08:14.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 4 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 client.114194.objecter _send_command 2 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 --> v2:172.21.15.143:6804/3538568827 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f15201212a0 con 0x7f150803e060 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f15177fe700 1 -- 172.21.15.5:0/1165330893 <== osd.5 v2:172.21.15.143:6804/3538568827 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1518003b80 con 0x7f150803e060 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f15177fe700 10 client.114194.objecter ms_dispatch 0x7f1520000fd0 command_reply(tid 2: 0 ) v1 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f15177fe700 10 client.114194.objecter _finish_command 2 = system:0 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f15177fe700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 4 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f15177fe700 15 client.114194.objecter _session_command_op_remove 5 2 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-16T11:08:14.729 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:08:14.730 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 librados: watch_flush enter 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 librados: watch_flush exit 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 client.114194.objecter close_session for osd.5 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6804/3538568827 conn(0x7f150803e060 msgr2=0x7f1508040500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6804/3538568827 conn(0x7f150803e060 0x7f1508040500 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter put_session s=0x7f150803d9b0 osd=5 1 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 client.114194.objecter shutdown clearing up homeless session... 2024-09-16T11:08:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 client.114194.objecter successfully canceled tick 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1508036f60 msgr2=0x7f1508039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 --2- 172.21.15.5:0/1165330893 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1508036f60 0x7f1508039420 secure :-1 s=READY pgs=3926 cs=0 l=1 rev1=1 crypto rx=0x7f15100018a0 tx=0x7f1510009040 comp rx=0 tx=0).stop 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 10 monclient: shutdown 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 msgr2=0x7f15201cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 --2- 172.21.15.5:0/1165330893 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f15200a37e0 0x7f15201cace0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f150c00c700 tx=0x7f150c00ca10 comp rx=0 tx=0).stop 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151f7fe700 1 -- 172.21.15.5:0/1165330893 reap_dead start 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 shutdown_connections 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 >> 172.21.15.5:0/1165330893 conn(0x7f15200a4df0 msgr2=0x7f15200a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 shutdown_connections 2024-09-16T11:08:14.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 -- 172.21.15.5:0/1165330893 wait complete. 2024-09-16T11:08:14.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.709+0000 7f151ffff700 1 librados: shutdown 2024-09-16T11:08:14.734 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T11:08:14.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: get_monmap_and_config 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: monmap: 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:14.854398+0000 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:14.854398+0000 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 Processor -- start 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 -- start start 2024-09-16T11:08:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: init 2024-09-16T11:08:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T11:08:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c0234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-b con 0x7f1c7c0b63f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c7c0b67d0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c7c0b6bf0 0x7f1c7c0b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c7c0b6bf0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7c0b6030 con 0x7f1c7c0b67d0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7c020000 con 0x7f1c7c0b6bf0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c01fce0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8359e700 10 monclient(hunting): authenticate will time out at 1953.431143s 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c81d9b700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1c7c0b67d0 0x7f1c7c0b9530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34600/0 (socket says 172.21.15.5:34600) 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c81d9b700 1 -- 172.21.15.5:0/2200824250 learned_addr learned my addr 172.21.15.5:0/2200824250 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 1 --2- 172.21.15.5:0/2200824250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c0234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): get_auth_request con 0x7f1c7c0b63f0 auth_method 0 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 1 --2- 172.21.15.5:0/2200824250 >> v2:172.21.15.143:3300/0 conn(0x7f1c7c0b6bf0 0x7f1c7c0b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): get_auth_request con 0x7f1c7c0b6bf0 auth_method 0 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 1 -- 172.21.15.5:0/2200824250 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325627681 0 0) 0x7f1c7c0b6030 con 0x7f1c7c0b67d0 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 10 monclient(hunting): _negotiate 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 10 monclient(hunting): my global_id is 74769 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8159a700 1 -- 172.21.15.5:0/2200824250 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c0035d0 con 0x7f1c7c0b67d0 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 10 monclient(hunting): handle_auth_done global_id 114197 payload 274 2024-09-16T11:08:14.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.849+0000 7f1c8259c700 1 -- 172.21.15.5:0/2200824250 >> v2:172.21.15.143:3300/0 conn(0x7f1c7c0b6bf0 msgr2=0x7f1c7c0b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 1 --2- 172.21.15.5:0/2200824250 >> v2:172.21.15.143:3300/0 conn(0x7f1c7c0b6bf0 0x7f1c7c0b9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 1 -- 172.21.15.5:0/2200824250 >> v1:172.21.15.5:6789/0 conn(0x7f1c7c0b67d0 legacy=0x7f1c7c0b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 1 monclient: found mon.noname-b 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 1 -- 172.21.15.5:0/2200824250 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c0222e0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8259c700 1 --2- 172.21.15.5:0/2200824250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c0234f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009850 tx=0x7f1c78009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ef9d82b8dd7c49e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 1 -- 172.21.15.5:0/2200824250 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78011830 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:14.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: dump: 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 1 -- 172.21.15.5:0/2200824250 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c78011990 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 5 monclient: authenticate success, global_id 114197 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 1 -- 172.21.15.5:0/2200824250 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78011d40 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 4 set_mon_vals no callback set 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8159a700 10 monclient: dump: 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 10 monclient: shutdown 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- 172.21.15.5:0/2200824250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 msgr2=0x7f1c7c0234f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 --2- 172.21.15.5:0/2200824250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c0234f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009850 tx=0x7f1c78009b60 comp rx=0 tx=0).stop 2024-09-16T11:08:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- 172.21.15.5:0/2200824250 shutdown_connections 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 --2- 172.21.15.5:0/2200824250 >> v2:172.21.15.143:3300/0 conn(0x7f1c7c0b6bf0 0x7f1c7c0b9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 --2- 172.21.15.5:0/2200824250 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c0234f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- 172.21.15.5:0/2200824250 >> 172.21.15.5:0/2200824250 conn(0x7f1c7c0064b0 msgr2=0x7f1c7c0068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- 172.21.15.5:0/2200824250 shutdown_connections 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- 172.21.15.5:0/2200824250 wait complete. 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 10 monclient: build_initial_monmap 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 10 monclient: monmap: 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 librados: starting msgr at 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 librados: starting objecter 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 Processor -- start 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 -- start start 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 librados: setting wanted keys 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 1 librados: calling monclient init 2024-09-16T11:08:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.853+0000 7f1c8359e700 10 monclient: init 2024-09-16T11:08:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-c con 0x7f1c7c0b63f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c0b67d0 0x7f1c7c14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-b con 0x7f1c7c0b67d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7c0b6bf0 0x7f1c7c0a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): picked mon.noname-a con 0x7f1c7c0b6bf0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c028b90 con 0x7f1c7c0b6bf0 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c0b9c70 con 0x7f1c7c0b67d0 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c020870 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient(hunting): authenticate will time out at 1953.437177s 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:53066/0 (socket says 172.21.15.5:53066) 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 -- 172.21.15.5:0/1559799217 learned_addr learned my addr 172.21.15.5:0/1559799217 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): get_auth_request con 0x7f1c7c0b63f0 auth_method 0 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7c0b6bf0 0x7f1c7c0a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): get_auth_request con 0x7f1c7c0b6bf0 auth_method 0 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c81d9b700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c0b67d0 0x7f1c7c14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c81d9b700 10 monclient(hunting): get_auth_request con 0x7f1c7c0b67d0 auth_method 0 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c81d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c81d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c81d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c82d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient(hunting): handle_auth_done global_id 114200 payload 824 2024-09-16T11:08:14.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 -- 172.21.15.5:0/1559799217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c0b67d0 msgr2=0x7f1c7c14ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c0b67d0 0x7f1c7c14ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 -- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7c0b6bf0 msgr2=0x7f1c7c0a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7c0b6bf0 0x7f1c7c0a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient: _finish_hunting 0 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 monclient: found mon.noname-c 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 -- 172.21.15.5:0/1559799217 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c780094b0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8259c700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c780090b0 tx=0x7f1c78004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 client.?.objecter ms_handle_connect 0x7f1c7c0b63f0 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:14.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78004300 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: dump: 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.880 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: _finish_auth 0 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 5 monclient: authenticate success, global_id 114200 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _renew_subs 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7801f070 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 -- 172.21.15.5:0/1559799217 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c0a6440 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c780036c0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c72ffd700 10 monclient: dump: 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c80d99700 4 set_mon_vals no callback set 2024-09-16T11:08:14.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 client.114200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _renew_subs 2024-09-16T11:08:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 -- 172.21.15.5:0/1559799217 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c0a4350 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c8359e700 1 librados: init done 2024-09-16T11:08:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c70ff9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c70ff9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c70ff9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.857+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5c005100 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f1c78021030 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 1 --2- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 0x7f1c64039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f1c78048ea0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 10 client.114200.objecter ms_dispatch 0x7f1c84000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 3 client.114200.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 3 client.114200.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 20 client.114200.objecter dump_active .. 0 homeless 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c81d9b700 1 --2- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 0x7f1c64039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c81d9b700 10 monclient: get_auth_request con 0x7f1c64036f60 auth_method 0 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c72ffd700 10 client.114200.objecter ms_handle_connect 0x7f1c64036f60 2024-09-16T11:08:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.861+0000 7f1c81d9b700 1 --2- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 0x7f1c64039420 secure :-1 s=READY pgs=3927 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0b5860 tx=0x7f1c74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:14.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.865+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1c780199a0 con 0x7f1c7c0b63f0 2024-09-16T11:08:14.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.865+0000 7f1c72ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:14.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:14.865+0000 7f1c72ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:15.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c5c000bc0 con 0x7f1c64036f60 2024-09-16T11:08:15.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c72ffd700 1 -- 172.21.15.5:0/1559799217 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c5c000bc0 con 0x7f1c64036f60 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 10 librados: watch_flush enter 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 10 librados: watch_flush exit 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 20 client.114200.objecter shutdown clearing up homeless session... 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 10 client.114200.objecter successfully canceled tick 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 msgr2=0x7f1c64039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 0x7f1c64039420 secure :-1 s=READY pgs=3927 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0b5860 tx=0x7f1c74009040 comp rx=0 tx=0).stop 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 10 monclient: shutdown 2024-09-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 msgr2=0x7f1c7c14e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.037+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c780090b0 tx=0x7f1c78004770 comp rx=0 tx=0).stop 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 shutdown_connections 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> v2:172.21.15.143:6812/1098205034 conn(0x7f1c64036f60 0x7f1c64039420 unknown :-1 s=CLOSED pgs=3927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1c7c0b6bf0 0x7f1c7c0a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c0b67d0 0x7f1c7c14ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 --2- 172.21.15.5:0/1559799217 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f1c7c0b63f0 0x7f1c7c14e720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 >> 172.21.15.5:0/1559799217 conn(0x7f1c7c0064b0 msgr2=0x7f1c7c0b8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 shutdown_connections 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 -- 172.21.15.5:0/1559799217 wait complete. 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.041+0000 7f1c70ff9700 1 librados: shutdown 2024-09-16T11:08:15.048 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T11:08:15.058 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:15.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 10 monclient: get_monmap_and_config 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 10 monclient: build_initial_monmap 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 10 monclient: monmap: 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:15.171846+0000 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:15.171846+0000 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:15.174 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 1 Processor -- start 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 1 -- start start 2024-09-16T11:08:15.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.165+0000 7f9776dea700 10 monclient: init 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97680295a0 0x7f9768023ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-c con 0x7f97680295a0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680248c0 0x7f9768024fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-b con 0x7f97680248c0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9768024400 0x7f9768025500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-a con 0x7f9768024400 addr v2:172.21.15.5:3300/0 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9768029980 con 0x7f9768024400 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9768026820 con 0x7f97680295a0 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9768027770 con 0x7f97680248c0 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:15.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient(hunting): authenticate will time out at 1953.748288s 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9768024400 0x7f9768025500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9768024400 0x7f9768025500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52002/0 (socket says 172.21.15.5:52002) 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680248c0 0x7f9768024fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 -- 172.21.15.5:0/1068349262 learned_addr learned my addr 172.21.15.5:0/1068349262 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): get_auth_request con 0x7f9768024400 auth_method 0 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): get_auth_request con 0x7f97680248c0 auth_method 0 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9775de8700 1 --2- 172.21.15.5:0/1068349262 >> v2:172.21.15.143:3300/0 conn(0x7f97680295a0 0x7f9768023ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9775de8700 10 monclient(hunting): get_auth_request con 0x7f97680295a0 auth_method 0 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9775de8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9775de8700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9775de8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient(hunting): handle_auth_done global_id 74775 payload 274 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 -- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680248c0 msgr2=0x7f9768024fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 --2- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680248c0 0x7f9768024fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 -- 172.21.15.5:0/1068349262 >> v2:172.21.15.143:3300/0 conn(0x7f97680295a0 msgr2=0x7f9768023ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 --2- 172.21.15.5:0/1068349262 >> v2:172.21.15.143:3300/0 conn(0x7f97680295a0 0x7f9768023ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient: _finish_hunting 0 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 monclient: found mon.noname-a 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 -- 172.21.15.5:0/1068349262 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768027a50 con 0x7f9768024400 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f97765e9700 1 --2- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768024400 0x7f9768025500 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f977004f940 tx=0x7f9770066b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8be8f2f8a6603c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:15.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 1 -- 172.21.15.5:0/1068349262 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770066bd0 con 0x7f9768024400 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: dump: 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: _finish_auth 0 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: _check_auth_tickets 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 1 -- 172.21.15.5:0/1068349262 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9770066e00 con 0x7f9768024400 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 5 monclient: authenticate success, global_id 74775 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 1 -- 172.21.15.5:0/1068349262 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97700008d0 con 0x7f9768024400 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 4 set_mon_vals no callback set 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9774de6700 10 monclient: dump: 2024-09-16T11:08:15.186 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:15.187 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 10 monclient: shutdown 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768024400 msgr2=0x7f9768025500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768024400 0x7f9768025500 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f977004f940 tx=0x7f9770066b60 comp rx=0 tx=0).stop 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- 172.21.15.5:0/1068349262 shutdown_connections 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9768024400 0x7f9768025500 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- 172.21.15.5:0/1068349262 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680248c0 0x7f9768024fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 --2- 172.21.15.5:0/1068349262 >> v2:172.21.15.143:3300/0 conn(0x7f97680295a0 0x7f9768023ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- 172.21.15.5:0/1068349262 >> 172.21.15.5:0/1068349262 conn(0x7f9768006440 msgr2=0x7f9768006850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- 172.21.15.5:0/1068349262 shutdown_connections 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.169+0000 7f9776dea700 1 -- 172.21.15.5:0/1068349262 wait complete. 2024-09-16T11:08:15.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient: build_initial_monmap 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient: monmap: 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 librados: starting msgr at 2024-09-16T11:08:15.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 librados: starting objecter 2024-09-16T11:08:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 Processor -- start 2024-09-16T11:08:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 -- start start 2024-09-16T11:08:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 librados: setting wanted keys 2024-09-16T11:08:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 librados: calling monclient init 2024-09-16T11:08:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient: init 2024-09-16T11:08:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T11:08:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768024400 0x7f9768150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-b con 0x7f9768024400 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:15.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-a con 0x7f97680295a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 0x7f97680a8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): picked mon.noname-c con 0x7f97680a7cc0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680ab2c0 con 0x7f97680295a0 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680a9800 con 0x7f9768024400 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680aa440 con 0x7f97680a7cc0 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:15.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9776dea700 10 monclient(hunting): authenticate will time out at 1953.755083s 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 0x7f97680a8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 0x7f97680a8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:53076/0 (socket says 172.21.15.5:53076) 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768024400 0x7f9768150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52016/0 (socket says 172.21.15.5:52016) 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 1 -- 172.21.15.5:0/1614610196 learned_addr learned my addr 172.21.15.5:0/1614610196 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): get_auth_request con 0x7f97680a7cc0 auth_method 0 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): get_auth_request con 0x7f97680295a0 auth_method 0 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): get_auth_request con 0x7f9768024400 auth_method 0 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97765e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:15.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f9775de8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 10 monclient(hunting): handle_auth_done global_id 74778 payload 824 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.173+0000 7f97755e7700 1 -- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 msgr2=0x7f97680a8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 0x7f97680a8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 -- 172.21.15.5:0/1614610196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768024400 msgr2=0x7f9768150940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768024400 0x7f9768150940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 10 monclient: _finish_hunting 0 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 monclient: found mon.noname-a 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 -- 172.21.15.5:0/1614610196 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9770069b60 con 0x7f97680295a0 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97755e7700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e910 tx=0x7f976c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 client.?.objecter ms_handle_connect 0x7f97680295a0 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:15.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c013680 con 0x7f97680295a0 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: dump: 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:15.201 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: _finish_auth 0 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: _check_auth_tickets 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 5 monclient: authenticate success, global_id 74778 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _renew_subs 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c0137e0 con 0x7f97680295a0 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976802d400 con 0x7f97680295a0 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c013b90 con 0x7f97680295a0 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:15.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 client.74778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 monclient: dump: 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9767fff700 4 set_mon_vals no callback set 2024-09-16T11:08:15.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _renew_subs 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9768024cb0 con 0x7f97680295a0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 1 librados: init done 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f976c013cf0 con 0x7f97680295a0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9768023ec0 con 0x7f97680295a0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 --2- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 0x7f9754039450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f976c047b80 con 0x7f97680295a0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 client.74778.objecter ms_dispatch 0x7f9770000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 3 client.74778.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:15.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 3 client.74778.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 20 client.74778.objecter dump_active .. 0 homeless 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9775de8700 1 --2- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 0x7f9754039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9775de8700 10 monclient: get_auth_request con 0x7f9754036f90 auth_method 0 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f97667fc700 10 client.74778.objecter ms_handle_connect 0x7f9754036f90 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.177+0000 7f9775de8700 1 --2- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 0x7f9754039450 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7f9760009980 tx=0x7f9760008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.185+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f976c049e60 con 0x7f97680295a0 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.185+0000 7f97667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:15.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.185+0000 7f97667fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:15.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.353+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f976802d290 con 0x7f9754036f90 2024-09-16T11:08:15.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f97667fc700 1 -- 172.21.15.5:0/1614610196 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133861 (secure 0 0 0) 0x7f976802d290 con 0x7f9754036f90 2024-09-16T11:08:15.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:15.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 10 librados: watch_flush enter 2024-09-16T11:08:15.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 10 librados: watch_flush exit 2024-09-16T11:08:15.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 20 client.74778.objecter shutdown clearing up homeless session... 2024-09-16T11:08:15.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 10 client.74778.objecter successfully canceled tick 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 msgr2=0x7f9754039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 0x7f9754039450 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7f9760009980 tx=0x7f9760008040 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 10 monclient: shutdown 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 msgr2=0x7f9768150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e910 tx=0x7f976c00ec20 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 shutdown_connections 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f97680a7cc0 0x7f97680a8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> v2:172.21.15.143:6812/1098205034 conn(0x7f9754036f90 0x7f9754039450 unknown :-1 s=CLOSED pgs=3928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97680295a0 0x7f9768150e80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 --2- 172.21.15.5:0/1614610196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9768024400 0x7f9768150940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:15.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 >> 172.21.15.5:0/1614610196 conn(0x7f9768006440 msgr2=0x7f9768021e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:15.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 shutdown_connections 2024-09-16T11:08:15.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 -- 172.21.15.5:0/1614610196 wait complete. 2024-09-16T11:08:15.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:15.369+0000 7f9776dea700 1 librados: shutdown 2024-09-16T11:08:15.379 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:15.392 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1236,"stamp":"2024-09-16T11:08:13.667794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":473120,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":444150,"kb_avail":565757920,"statfs":{"total":579820584960,"available":579336110080,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":454810278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003952"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":663,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104625+0000","last_change":"2024-09-16T10:51:10.895648+0000","last_active":"2024-09-16T11:07:01.104625+0000","last_peered":"2024-09-16T11:07:01.104625+0000","last_clean":"2024-09-16T11:07:01.104625+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:07:01.104625+0000","last_undegraded":"2024-09-16T11:07:01.104625+0000","last_fullsized":"2024-09-16T11:07:01.104625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:14.539597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":636,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221962+0000","last_change":"2024-09-16T10:51:10.811863+0000","last_active":"2024-09-16T11:07:00.221962+0000","last_peered":"2024-09-16T11:07:00.221962+0000","last_clean":"2024-09-16T11:07:00.221962+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:07:00.221962+0000","last_undegraded":"2024-09-16T11:07:00.221962+0000","last_fullsized":"2024-09-16T11:07:00.221962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:51.833056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053499+0000","last_change":"2024-09-16T10:51:10.836991+0000","last_active":"2024-09-16T11:07:00.053499+0000","last_peered":"2024-09-16T11:07:00.053499+0000","last_clean":"2024-09-16T11:07:00.053499+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:07:00.053499+0000","last_undegraded":"2024-09-16T11:07:00.053499+0000","last_fullsized":"2024-09-16T11:07:00.053499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:54.839408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.870842+0000","last_change":"2024-09-16T10:51:10.861754+0000","last_active":"2024-09-16T11:07:01.870842+0000","last_peered":"2024-09-16T11:07:01.870842+0000","last_clean":"2024-09-16T11:07:01.870842+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:07:01.870842+0000","last_undegraded":"2024-09-16T11:07:01.870842+0000","last_fullsized":"2024-09-16T11:07:01.870842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:10.098818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054257+0000","last_change":"2024-09-16T10:51:10.888616+0000","last_active":"2024-09-16T11:07:00.054257+0000","last_peered":"2024-09-16T11:07:00.054257+0000","last_clean":"2024-09-16T11:07:00.054257+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:07:00.054257+0000","last_undegraded":"2024-09-16T11:07:00.054257+0000","last_fullsized":"2024-09-16T11:07:00.054257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:22:47.053233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551276+0000","last_change":"2024-09-16T10:51:10.820735+0000","last_active":"2024-09-16T11:07:00.551276+0000","last_peered":"2024-09-16T11:07:00.551276+0000","last_clean":"2024-09-16T11:07:00.551276+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:07:00.551276+0000","last_undegraded":"2024-09-16T11:07:00.551276+0000","last_fullsized":"2024-09-16T11:07:00.551276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:57:00.585239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":650,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.053844+0000","last_change":"2024-09-16T10:51:10.906082+0000","last_active":"2024-09-16T11:07:00.053844+0000","last_peered":"2024-09-16T11:07:00.053844+0000","last_clean":"2024-09-16T11:07:00.053844+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:07:00.053844+0000","last_undegraded":"2024-09-16T11:07:00.053844+0000","last_fullsized":"2024-09-16T11:07:00.053844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:24.987446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":646,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051232+0000","last_change":"2024-09-16T10:51:10.825984+0000","last_active":"2024-09-16T11:07:00.051232+0000","last_peered":"2024-09-16T11:07:00.051232+0000","last_clean":"2024-09-16T11:07:00.051232+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:07:00.051232+0000","last_undegraded":"2024-09-16T11:07:00.051232+0000","last_fullsized":"2024-09-16T11:07:00.051232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:31:17.054426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":632,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551587+0000","last_change":"2024-09-16T10:51:10.851590+0000","last_active":"2024-09-16T11:07:00.551587+0000","last_peered":"2024-09-16T11:07:00.551587+0000","last_clean":"2024-09-16T11:07:00.551587+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:07:00.551587+0000","last_undegraded":"2024-09-16T11:07:00.551587+0000","last_fullsized":"2024-09-16T11:07:00.551587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:45:52.807901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":637,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221819+0000","last_change":"2024-09-16T10:51:10.818746+0000","last_active":"2024-09-16T11:07:00.221819+0000","last_peered":"2024-09-16T11:07:00.221819+0000","last_clean":"2024-09-16T11:07:00.221819+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:07:00.221819+0000","last_undegraded":"2024-09-16T11:07:00.221819+0000","last_fullsized":"2024-09-16T11:07:00.221819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.511253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961018+0000","last_change":"2024-09-16T10:51:10.916719+0000","last_active":"2024-09-16T11:07:01.961018+0000","last_peered":"2024-09-16T11:07:01.961018+0000","last_clean":"2024-09-16T11:07:01.961018+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:07:01.961018+0000","last_undegraded":"2024-09-16T11:07:01.961018+0000","last_fullsized":"2024-09-16T11:07:01.961018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:53:07.031545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":647,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959927+0000","last_change":"2024-09-16T10:51:10.807192+0000","last_active":"2024-09-16T11:07:01.959927+0000","last_peered":"2024-09-16T11:07:01.959927+0000","last_clean":"2024-09-16T11:07:01.959927+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:07:01.959927+0000","last_undegraded":"2024-09-16T11:07:01.959927+0000","last_fullsized":"2024-09-16T11:07:01.959927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:36:17.091636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958919+0000","last_change":"2024-09-16T10:51:10.888609+0000","last_active":"2024-09-16T11:07:01.958919+0000","last_peered":"2024-09-16T11:07:01.958919+0000","last_clean":"2024-09-16T11:07:01.958919+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:07:01.958919+0000","last_undegraded":"2024-09-16T11:07:01.958919+0000","last_fullsized":"2024-09-16T11:07:01.958919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:51:59.284621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233208+0000","last_change":"2024-09-16T10:50:03.526459+0000","last_active":"2024-09-16T11:07:01.233208+0000","last_peered":"2024-09-16T11:07:01.233208+0000","last_clean":"2024-09-16T11:07:01.233208+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:07:01.233208+0000","last_undegraded":"2024-09-16T11:07:01.233208+0000","last_fullsized":"2024-09-16T11:07:01.233208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:38:44.255560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961200+0000","last_change":"2024-09-16T10:51:10.907339+0000","last_active":"2024-09-16T11:07:01.961200+0000","last_peered":"2024-09-16T11:07:01.961200+0000","last_clean":"2024-09-16T11:07:01.961200+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:07:01.961200+0000","last_undegraded":"2024-09-16T11:07:01.961200+0000","last_fullsized":"2024-09-16T11:07:01.961200+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:03:22.367946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959643+0000","last_change":"2024-09-16T10:50:03.429266+0000","last_active":"2024-09-16T11:07:01.959643+0000","last_peered":"2024-09-16T11:07:01.959643+0000","last_clean":"2024-09-16T11:07:01.959643+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:07:01.959643+0000","last_undegraded":"2024-09-16T11:07:01.959643+0000","last_fullsized":"2024-09-16T11:07:01.959643+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":554,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051532+0000","last_change":"2024-09-16T10:52:56.726039+0000","last_active":"2024-09-16T11:07:00.051532+0000","last_peered":"2024-09-16T11:07:00.051532+0000","last_clean":"2024-09-16T11:07:00.051532+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:07:00.051532+0000","last_undegraded":"2024-09-16T11:07:00.051532+0000","last_fullsized":"2024-09-16T11:07:00.051532+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:50:38.112159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":648,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221678+0000","last_change":"2024-09-16T10:50:03.526540+0000","last_active":"2024-09-16T11:07:00.221678+0000","last_peered":"2024-09-16T11:07:00.221678+0000","last_clean":"2024-09-16T11:07:00.221678+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:07:00.221678+0000","last_undegraded":"2024-09-16T11:07:00.221678+0000","last_fullsized":"2024-09-16T11:07:00.221678+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:28.696426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.961095+0000","last_change":"2024-09-16T10:51:10.915110+0000","last_active":"2024-09-16T11:07:01.961095+0000","last_peered":"2024-09-16T11:07:01.961095+0000","last_clean":"2024-09-16T11:07:01.961095+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:07:01.961095+0000","last_undegraded":"2024-09-16T11:07:01.961095+0000","last_fullsized":"2024-09-16T11:07:01.961095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:58:54.203725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":658,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.051747+0000","last_change":"2024-09-16T10:50:03.429452+0000","last_active":"2024-09-16T11:07:00.051747+0000","last_peered":"2024-09-16T11:07:00.051747+0000","last_clean":"2024-09-16T11:07:00.051747+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:07:00.051747+0000","last_undegraded":"2024-09-16T11:07:00.051747+0000","last_fullsized":"2024-09-16T11:07:00.051747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:05:54.422123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":677,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104372+0000","last_change":"2024-09-16T10:50:03.429367+0000","last_active":"2024-09-16T11:07:01.104372+0000","last_peered":"2024-09-16T11:07:01.104372+0000","last_clean":"2024-09-16T11:07:01.104372+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:07:01.104372+0000","last_undegraded":"2024-09-16T11:07:01.104372+0000","last_fullsized":"2024-09-16T11:07:01.104372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:26.377288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958388+0000","last_change":"2024-09-16T10:51:10.900283+0000","last_active":"2024-09-16T11:07:01.958388+0000","last_peered":"2024-09-16T11:07:01.958388+0000","last_clean":"2024-09-16T11:07:01.958388+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:07:01.958388+0000","last_undegraded":"2024-09-16T11:07:01.958388+0000","last_fullsized":"2024-09-16T11:07:01.958388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:51.095223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":644,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551103+0000","last_change":"2024-09-16T10:50:03.688026+0000","last_active":"2024-09-16T11:07:00.551103+0000","last_peered":"2024-09-16T11:07:00.551103+0000","last_clean":"2024-09-16T11:07:00.551103+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:07:00.551103+0000","last_undegraded":"2024-09-16T11:07:00.551103+0000","last_fullsized":"2024-09-16T11:07:00.551103+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:33:35.069992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959213+0000","last_change":"2024-09-16T10:51:10.777356+0000","last_active":"2024-09-16T11:07:01.959213+0000","last_peered":"2024-09-16T11:07:01.959213+0000","last_clean":"2024-09-16T11:07:01.959213+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:07:01.959213+0000","last_undegraded":"2024-09-16T11:07:01.959213+0000","last_fullsized":"2024-09-16T11:07:01.959213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:54.038836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":651,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232503+0000","last_change":"2024-09-16T10:50:03.526951+0000","last_active":"2024-09-16T11:07:01.232503+0000","last_peered":"2024-09-16T11:07:01.232503+0000","last_clean":"2024-09-16T11:07:01.232503+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:07:01.232503+0000","last_undegraded":"2024-09-16T11:07:01.232503+0000","last_fullsized":"2024-09-16T11:07:01.232503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:51.670080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232530+0000","last_change":"2024-09-16T10:51:10.824359+0000","last_active":"2024-09-16T11:07:01.232530+0000","last_peered":"2024-09-16T11:07:01.232530+0000","last_clean":"2024-09-16T11:07:01.232530+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:07:01.232530+0000","last_undegraded":"2024-09-16T11:07:01.232530+0000","last_fullsized":"2024-09-16T11:07:01.232530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:09.821124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":676,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.104414+0000","last_change":"2024-09-16T10:50:03.428485+0000","last_active":"2024-09-16T11:07:01.104414+0000","last_peered":"2024-09-16T11:07:01.104414+0000","last_clean":"2024-09-16T11:07:01.104414+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:07:01.104414+0000","last_undegraded":"2024-09-16T11:07:01.104414+0000","last_fullsized":"2024-09-16T11:07:01.104414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:50:00.374573+0000","last_clean_scrub_stamp":"2024-09-16T10:50:00.374573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:36.593597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":797,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232630+0000","last_change":"2024-09-16T10:50:00.397436+0000","last_active":"2024-09-16T11:07:01.232630+0000","last_peered":"2024-09-16T11:07:01.232630+0000","last_clean":"2024-09-16T11:07:01.232630+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:07:01.232630+0000","last_undegraded":"2024-09-16T11:07:01.232630+0000","last_fullsized":"2024-09-16T11:07:01.232630+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:49:59.368051+0000","last_clean_scrub_stamp":"2024-09-16T10:49:59.368051+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:55.979367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232619+0000","last_change":"2024-09-16T10:51:10.819816+0000","last_active":"2024-09-16T11:07:01.232619+0000","last_peered":"2024-09-16T11:07:01.232619+0000","last_clean":"2024-09-16T11:07:01.232619+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:07:01.232619+0000","last_undegraded":"2024-09-16T11:07:01.232619+0000","last_fullsized":"2024-09-16T11:07:01.232619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:01.570513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232841+0000","last_change":"2024-09-16T10:51:10.881698+0000","last_active":"2024-09-16T11:07:01.232841+0000","last_peered":"2024-09-16T11:07:01.232841+0000","last_clean":"2024-09-16T11:07:01.232841+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:07:01.232841+0000","last_undegraded":"2024-09-16T11:07:01.232841+0000","last_fullsized":"2024-09-16T11:07:01.232841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:17.134723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":646,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959641+0000","last_change":"2024-09-16T10:51:10.823091+0000","last_active":"2024-09-16T11:07:01.959641+0000","last_peered":"2024-09-16T11:07:01.959641+0000","last_clean":"2024-09-16T11:07:01.959641+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:07:01.959641+0000","last_undegraded":"2024-09-16T11:07:01.959641+0000","last_fullsized":"2024-09-16T11:07:01.959641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:01:32.535561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":631,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.551175+0000","last_change":"2024-09-16T10:51:10.890050+0000","last_active":"2024-09-16T11:07:00.551175+0000","last_peered":"2024-09-16T11:07:00.551175+0000","last_clean":"2024-09-16T11:07:00.551175+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:07:00.551175+0000","last_undegraded":"2024-09-16T11:07:00.551175+0000","last_fullsized":"2024-09-16T11:07:00.551175+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:00.950193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":649,"reported_epoch":692,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.054067+0000","last_change":"2024-09-16T10:51:10.851438+0000","last_active":"2024-09-16T11:07:00.054067+0000","last_peered":"2024-09-16T11:07:00.054067+0000","last_clean":"2024-09-16T11:07:00.054067+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:07:00.054067+0000","last_undegraded":"2024-09-16T11:07:00.054067+0000","last_fullsized":"2024-09-16T11:07:00.054067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:50:26.462001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":539,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871448+0000","last_change":"2024-09-16T10:52:56.729849+0000","last_active":"2024-09-16T11:07:01.871448+0000","last_peered":"2024-09-16T11:07:01.871448+0000","last_clean":"2024-09-16T11:07:01.871448+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:07:01.871448+0000","last_undegraded":"2024-09-16T11:07:01.871448+0000","last_fullsized":"2024-09-16T11:07:01.871448+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:09.227854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":635,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-09-16T11:07:00.221534+0000","last_change":"2024-09-16T10:51:10.773626+0000","last_active":"2024-09-16T11:07:00.221534+0000","last_peered":"2024-09-16T11:07:00.221534+0000","last_clean":"2024-09-16T11:07:00.221534+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:07:00.221534+0000","last_undegraded":"2024-09-16T11:07:00.221534+0000","last_fullsized":"2024-09-16T11:07:00.221534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:31.797539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":631,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.871689+0000","last_change":"2024-09-16T10:51:10.895063+0000","last_active":"2024-09-16T11:07:01.871689+0000","last_peered":"2024-09-16T11:07:01.871689+0000","last_clean":"2024-09-16T11:07:01.871689+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:07:01.871689+0000","last_undegraded":"2024-09-16T11:07:01.871689+0000","last_fullsized":"2024-09-16T11:07:01.871689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:59.114090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":636,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233205+0000","last_change":"2024-09-16T10:51:10.830383+0000","last_active":"2024-09-16T11:07:01.233205+0000","last_peered":"2024-09-16T11:07:01.233205+0000","last_clean":"2024-09-16T11:07:01.233205+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:07:01.233205+0000","last_undegraded":"2024-09-16T11:07:01.233205+0000","last_fullsized":"2024-09-16T11:07:01.233205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:39.706733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":633,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958113+0000","last_change":"2024-09-16T10:51:10.849321+0000","last_active":"2024-09-16T11:07:01.958113+0000","last_peered":"2024-09-16T11:07:01.958113+0000","last_clean":"2024-09-16T11:07:01.958113+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:07:01.958113+0000","last_undegraded":"2024-09-16T11:07:01.958113+0000","last_fullsized":"2024-09-16T11:07:01.958113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:42.862278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"20'7","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.959504+0000","last_change":"2024-09-16T10:51:10.808306+0000","last_active":"2024-09-16T11:07:01.959504+0000","last_peered":"2024-09-16T11:07:01.959504+0000","last_clean":"2024-09-16T11:07:01.959504+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:07:01.959504+0000","last_undegraded":"2024-09-16T11:07:01.959504+0000","last_fullsized":"2024-09-16T11:07:01.959504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:57:18.223681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":664,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.960466+0000","last_change":"2024-09-16T10:51:10.869585+0000","last_active":"2024-09-16T11:07:01.960466+0000","last_peered":"2024-09-16T11:07:01.960466+0000","last_clean":"2024-09-16T11:07:01.960466+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:07:01.960466+0000","last_undegraded":"2024-09-16T11:07:01.960466+0000","last_fullsized":"2024-09-16T11:07:01.960466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:37.869026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":638,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.232719+0000","last_change":"2024-09-16T10:51:10.814816+0000","last_active":"2024-09-16T11:07:01.232719+0000","last_peered":"2024-09-16T11:07:01.232719+0000","last_clean":"2024-09-16T11:07:01.232719+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:07:01.232719+0000","last_undegraded":"2024-09-16T11:07:01.232719+0000","last_fullsized":"2024-09-16T11:07:01.232719+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:29:46.969757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80048,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291792,"statfs":{"total":96636764160,"available":96554795008,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.60299999999999998,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":1.123,"15min":1.984},"last":0.73099999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.80800000000000005},"min":{"1min":0.65800000000000003,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.3},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.74399999999999999,"5min":0.56999999999999995,"15min":0.53000000000000003},"max":{"1min":1.1459999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.75900000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.68100000000000005,"5min":0.54200000000000004,"15min":0.45400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.1779999999999999},"last":0.624},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":1.0620000000000001,"15min":1.22},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67600000000000005,"15min":0.73199999999999998},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.385},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82699999999999996,"15min":0.78900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.365},"last":0.83199999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.72299999999999998,"15min":0.71199999999999997},"min":{"1min":0.58999999999999997,"5min":0.53800000000000003,"15min":0.39400000000000002},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95399999999999996,"15min":0.92900000000000005},"min":{"1min":0.83399999999999996,"5min":0.72799999999999998,"15min":0.499},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.91800000000000004}]}]},{"osd":4,"up_from":11,"seq":47244640476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80040,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291800,"statfs":{"total":96636764160,"available":96554803200,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.78600000000000003,"15min":0.82899999999999996},"min":{"1min":0.55800000000000005,"5min":0.53100000000000003,"15min":0.36599999999999999},"max":{"1min":1.169,"5min":1.169,"15min":5.351},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80000000000000004,"15min":0.78000000000000003},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.29999999999999999},"max":{"1min":1.032,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90800000000000003,"15min":0.91800000000000004},"min":{"1min":0.623,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.268,"5min":1.268,"15min":10.134},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.91900000000000004,"15min":0.91200000000000003},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.0980000000000001,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.81999999999999995}]},{"osd":2,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.83499999999999996,"15min":0.89300000000000002},"min":{"1min":0.625,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.93999999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.68000000000000005,"5min":0.61099999999999999,"15min":0.46899999999999997},"max":{"1min":1.044,"5min":1.28,"15min":1.3859999999999999},"last":0.91500000000000004}]},{"osd":3,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68300000000000005,"15min":0.73499999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.34100000000000003},"max":{"1min":0.86199999999999999,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.752,"15min":0.77600000000000002},"min":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.44500000000000001},"max":{"1min":0.996,"5min":1.268,"15min":5.0579999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Sep 16 11:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.75900000000000001,"15min":0.77000000000000002},"min":{"1min":0.56599999999999995,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.996,"5min":1.153,"15min":3.7770000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78400000000000003,"15min":0.83499999999999996},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":1.111,"15min":8.5559999999999992},"last":0.68300000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76476,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295364,"statfs":{"total":96636764160,"available":96558452736,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.37},"max":{"1min":0.97799999999999998,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.35599999999999998},"max":{"1min":0.93400000000000005,"5min":1.1000000000000001,"15min":1.609},"last":0.72499999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.871,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.436},"max":{"1min":1.01,"5min":1.4430000000000001,"15min":1.821},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.85499999999999998,"5min":0.64500000000000002,"15min":0.46899999999999997},"max":{"1min":1.101,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.84099999999999997}]},{"osd":2,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.67700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98899999999999999,"5min":1.2290000000000001,"15min":10.129},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.65400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.006,"5min":2.3559999999999999,"15min":10.449999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":3.4710000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.47999999999999998,"15min":0.33500000000000002},"max":{"1min":0.98399999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.749,"15min":0.77000000000000002},"min":{"1min":0.56200000000000006,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":1.179,"5min":1.1870000000000001,"15min":1.353},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81200000000000006,"15min":0.84199999999999997},"min":{"1min":0.76000000000000001,"5min":0.52200000000000002,"15min":0.39600000000000002},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":2.625},"last":0.75900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640476,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71848,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299992,"statfs":{"total":96636764160,"available":96563191808,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.76800000000000002,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.39500000000000002},"max":{"1min":0.94899999999999995,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55900000000000005,"15min":0.57199999999999995},"min":{"1min":0.438,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.65300000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.76900000000000002,"15min":0.746},"min":{"1min":0.63200000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.91300000000000003,"5min":1.155,"15min":1.155},"last":0.72999999999999998}]},{"osd":3,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86099999999999999,"15min":0.86299999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0569999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.89400000000000002},"min":{"1min":0.71599999999999997,"5min":0.66100000000000003,"15min":0.60099999999999998},"max":{"1min":1.1739999999999999,"5min":1.341,"15min":1.774},"last":0.97499999999999998}]},{"osd":4,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.88700000000000001,"15min":0.89900000000000002},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.59499999999999997},"max":{"1min":1.0720000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90100000000000002,"15min":0.875},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59899999999999998},"max":{"1min":1.097,"5min":1.1120000000000001,"15min":1.7090000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Mon Sep 16 11:07:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.88300000000000001,"15min":0.88100000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.141,"5min":1.397,"15min":3.7989999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.95499999999999996,"15min":0.95099999999999996},"min":{"1min":0.81000000000000005,"5min":0.76700000000000002,"15min":0.57799999999999996},"max":{"1min":1.1950000000000001,"5min":1.204,"15min":3.2799999999999998},"last":0.84099999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84144,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":79486,"kb_avail":94287696,"statfs":{"total":96636764160,"available":96550600704,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81394119},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.72699999999999998},"min":{"1min":0.59499999999999997,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":1.04,"15min":1.845},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.51900000000000002,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":1.034,"5min":1.034,"15min":4.9260000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.89700000000000002,"15min":2.8370000000000002},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.879,"15min":0.89200000000000002},"min":{"1min":0.76400000000000001,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.95799999999999996,"15min":0.96799999999999997},"min":{"1min":0.746,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.208,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.89700000000000002},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.93899999999999995,"5min":1.2729999999999999,"15min":2.802},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94899999999999995,"15min":0.94199999999999995},"min":{"1min":0.76700000000000002,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.206,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.90800000000000003},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.71299999999999997,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.159,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.69399999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80564,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291276,"statfs":{"total":96636764160,"available":96554266624,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.221},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":2.6469999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.247},"max":{"1min":1.079,"5min":1.079,"15min":2.3479999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71499999999999997,"5min":1.0289999999999999,"15min":1.327},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":1.228,"15min":4.1319999999999997},"last":0.75700000000000001}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85799999999999998,"15min":0.90300000000000002},"min":{"1min":0.69399999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.70699999999999996,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":1.175,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.88700000000000001}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.88500000000000001,"15min":0.91700000000000004},"min":{"1min":0.73099999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.008,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.74299999999999999,"5min":0.46999999999999997,"15min":0.36699999999999999},"max":{"1min":1.1020000000000001,"5min":1.302,"15min":3.1440000000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.89200000000000002},"min":{"1min":0.69399999999999995,"5min":0.49399999999999999,"15min":0.45200000000000001},"max":{"1min":1.1479999999999999,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88,"15min":0.94499999999999995},"min":{"1min":0.69399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":1.0409999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.92600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:15.401 INFO:tasks.ceph:pgid 19.f last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.401 INFO:tasks.ceph:pgid 19.e last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.401 INFO:tasks.ceph:pgid 19.d last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.402 INFO:tasks.ceph:pgid 19.c last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.402 INFO:tasks.ceph:pgid 19.b last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.402 INFO:tasks.ceph:pgid 19.a last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.402 INFO:tasks.ceph:pgid 19.9 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.402 INFO:tasks.ceph:pgid 19.8 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.403 INFO:tasks.ceph:pgid 19.7 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.403 INFO:tasks.ceph:pgid 19.6 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.403 INFO:tasks.ceph:pgid 19.5 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.403 INFO:tasks.ceph:pgid 19.4 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.403 INFO:tasks.ceph:pgid 19.16 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 19.17 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 19.14 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.404 INFO:tasks.ceph:pgid 19.15 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.405 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.405 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.405 INFO:tasks.ceph:pgid 19.13 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.405 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.405 INFO:tasks.ceph:pgid 19.10 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.406 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.406 INFO:tasks.ceph:pgid 19.11 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.406 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T10:50:00.374573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=50, tm_sec=0, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.406 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T10:49:59.368051+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=49, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.407 INFO:tasks.ceph:pgid 19.12 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.407 INFO:tasks.ceph:pgid 19.18 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.407 INFO:tasks.ceph:pgid 19.19 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.407 INFO:tasks.ceph:pgid 19.1a last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.407 INFO:tasks.ceph:pgid 19.1b last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.408 INFO:tasks.ceph:pgid 19.1c last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.408 INFO:tasks.ceph:pgid 19.1d last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.408 INFO:tasks.ceph:pgid 19.1e last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.408 INFO:tasks.ceph:pgid 19.1f last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.408 INFO:tasks.ceph:pgid 19.0 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.409 INFO:tasks.ceph:pgid 19.1 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.409 INFO:tasks.ceph:pgid 19.2 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.410 INFO:tasks.ceph:pgid 19.3 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:15.410 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:08:35.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:35.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: get_monmap_and_config 2024-09-16T11:08:35.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: build_initial_monmap 2024-09-16T11:08:35.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: monmap: 2024-09-16T11:08:35.536 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:35.534822+0000 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:35.534822+0000 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:35.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 Processor -- start 2024-09-16T11:08:35.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 -- start start 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: init 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f218409b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-b con 0x7f218409b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f218409bd00 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f218409c0e0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218409fd30 con 0x7f218409c0e0 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21841334d0 con 0x7f218409bd00 2024-09-16T11:08:35.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218409c570 con 0x7f218409b3e0 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218b10a700 10 monclient(hunting): authenticate will time out at 1974.111452s 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a108700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f218409b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a909700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f218409c0e0 0x7f2184132dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36416/0 (socket says 172.21.15.5:36416) 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a909700 1 -- 172.21.15.5:0/3265217317 learned_addr learned my addr 172.21.15.5:0/3265217317 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a108700 10 monclient(hunting): get_auth_request con 0x7f218409b3e0 auth_method 0 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a108700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a108700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.529+0000 7f218a108700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026151177 0 0) 0x7f218409fd30 con 0x7f218409c0e0 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _negotiate 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): my global_id is 74781 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21700035d0 con 0x7f218409c0e0 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113254128 0 0) 0x7f21841334d0 con 0x7f218409bd00 2024-09-16T11:08:35.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _negotiate 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient(hunting): my global_id is 94462 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f218409fd30 con 0x7f218409bd00 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient(hunting): handle_auth_done global_id 114209 payload 274 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 1 -- 172.21.15.5:0/3265217317 >> v1:172.21.15.143:6789/0 conn(0x7f218409bd00 legacy=0x7f21841326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 1 -- 172.21.15.5:0/3265217317 >> v1:172.21.15.5:6789/0 conn(0x7f218409c0e0 legacy=0x7f2184132dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient: _finish_hunting 0 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 1 monclient: found mon.noname-b 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 1 -- 172.21.15.5:0/3265217317 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218409e350 con 0x7f218409b3e0 2024-09-16T11:08:35.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218a108700 1 --2- 172.21.15.5:0/3265217317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f218409b7c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2180009880 tx=0x7f2180009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa879394abb86289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180011830 con 0x7f218409b3e0 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: dump: 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: _finish_auth 0 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: _check_auth_tickets 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180011990 con 0x7f218409b3e0 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 5 monclient: authenticate success, global_id 114209 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 1 -- 172.21.15.5:0/3265217317 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180011d00 con 0x7f218409b3e0 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f2189106700 10 monclient: dump: 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:35.547 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 4 set_mon_vals no callback set 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 10 monclient: shutdown 2024-09-16T11:08:35.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 -- 172.21.15.5:0/3265217317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 msgr2=0x7f218409b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 --2- 172.21.15.5:0/3265217317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f218409b7c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2180009880 tx=0x7f2180009b90 comp rx=0 tx=0).stop 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 -- 172.21.15.5:0/3265217317 shutdown_connections 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 --2- 172.21.15.5:0/3265217317 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f218409b7c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 -- 172.21.15.5:0/3265217317 >> 172.21.15.5:0/3265217317 conn(0x7f21840a45d0 msgr2=0x7f21840a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 -- 172.21.15.5:0/3265217317 shutdown_connections 2024-09-16T11:08:35.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.533+0000 7f218b10a700 1 -- 172.21.15.5:0/3265217317 wait complete. 2024-09-16T11:08:35.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient: build_initial_monmap 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient: monmap: 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:35.555 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 librados: starting msgr at 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 librados: starting objecter 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 Processor -- start 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 -- start start 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 librados: setting wanted keys 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 librados: calling monclient init 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient: init 2024-09-16T11:08:35.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-c con 0x7f218409b3e0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f218409bd00 0x7f21841c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-b con 0x7f218409bd00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f218409c0e0 0x7f2184120ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): picked mon.noname-a con 0x7f218409c0e0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2184124870 con 0x7f218409c0e0 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218409b160 con 0x7f218409bd00 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:35.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2184124ac0 con 0x7f218409b3e0 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:54452/0 (socket says 172.21.15.5:54452) 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 -- 172.21.15.5:0/3921660891 learned_addr learned my addr 172.21.15.5:0/3921660891 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218b10a700 10 monclient(hunting): authenticate will time out at 1974.119340s 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): get_auth_request con 0x7f218409b3e0 auth_method 0 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f218409c0e0 0x7f2184120ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): get_auth_request con 0x7f218409c0e0 auth_method 0 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f2189907700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f218409bd00 0x7f21841c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a909700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f2189907700 10 monclient(hunting): get_auth_request con 0x7f218409bd00 auth_method 0 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f2189907700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f2189907700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f2189907700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient(hunting): handle_auth_done global_id 114212 payload 824 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 -- 172.21.15.5:0/3921660891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f218409bd00 msgr2=0x7f21841c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f218409bd00 0x7f21841c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 -- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f218409c0e0 msgr2=0x7f2184120ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f218409c0e0 0x7f2184120ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient: _finish_hunting 0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 monclient: found mon.noname-c 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.537+0000 7f218a108700 1 -- 172.21.15.5:0/3921660891 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21800094b0 con 0x7f218409b3e0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218a108700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21800097d0 tx=0x7f2180004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 client.?.objecter ms_handle_connect 0x7f218409b3e0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21800038c0 con 0x7f218409b3e0 2024-09-16T11:08:35.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: dump: 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: _finish_auth 0 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: _check_auth_tickets 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 5 monclient: authenticate success, global_id 114212 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180003a20 con 0x7f218409b3e0 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180003dd0 con 0x7f218409b3e0 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _renew_subs 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2184070820 con 0x7f218409b3e0 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f2188905700 4 set_mon_vals no callback set 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 monclient: dump: 2024-09-16T11:08:35.562 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:35.563 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 client.114212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _renew_subs 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2184122e80 con 0x7f218409b3e0 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 1 librados: init done 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f218001b400 con 0x7f218409b3e0 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2184124620 con 0x7f218409b3e0 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 --2- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 0x7f2168039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:35.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f2180049390 con 0x7f218409b3e0 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 client.114212.objecter ms_dispatch 0x7f2184000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 3 client.114212.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 3 client.114212.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 20 client.114212.objecter dump_active .. 0 homeless 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f2189907700 1 --2- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 0x7f2168039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f2189907700 10 monclient: get_auth_request con 0x7f2168036f60 auth_method 0 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f2189907700 1 --2- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 0x7f2168039420 secure :-1 s=READY pgs=3929 cs=0 l=1 rev1=1 crypto rx=0x7f21840a06c0 tx=0x7f2174009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:35.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.541+0000 7f217affd700 10 client.114212.objecter ms_handle_connect 0x7f2168036f60 2024-09-16T11:08:35.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.549+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2180006ad0 con 0x7f218409b3e0 2024-09-16T11:08:35.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.549+0000 7f217affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:35.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.549+0000 7f217affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:35.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.717+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f218409cfe0 con 0x7f2168036f60 2024-09-16T11:08:35.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.729+0000 7f217affd700 1 -- 172.21.15.5:0/3921660891 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134551 (secure 0 0 0) 0x7f21740050d0 con 0x7f2168036f60 2024-09-16T11:08:35.736 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:35.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.729+0000 7f218b10a700 10 librados: watch_flush enter 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.729+0000 7f218b10a700 10 librados: watch_flush exit 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.729+0000 7f218b10a700 20 client.114212.objecter shutdown clearing up homeless session... 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.729+0000 7f218b10a700 10 client.114212.objecter successfully canceled tick 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 msgr2=0x7f2168039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 0x7f2168039420 secure :-1 s=READY pgs=3929 cs=0 l=1 rev1=1 crypto rx=0x7f21840a06c0 tx=0x7f2174009040 comp rx=0 tx=0).stop 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 10 monclient: shutdown 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 msgr2=0x7f21841c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21800097d0 tx=0x7f2180004770 comp rx=0 tx=0).stop 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 shutdown_connections 2024-09-16T11:08:35.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> v2:172.21.15.143:6812/1098205034 conn(0x7f2168036f60 0x7f2168039420 unknown :-1 s=CLOSED pgs=3929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f218409c0e0 0x7f2184120ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f218409bd00 0x7f21841c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 --2- 172.21.15.5:0/3921660891 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f218409b3e0 0x7f21841c8740 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 >> 172.21.15.5:0/3921660891 conn(0x7f21840a45d0 msgr2=0x7f2184070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 shutdown_connections 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 -- 172.21.15.5:0/3921660891 wait complete. 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:35.733+0000 7f218b10a700 1 librados: shutdown 2024-09-16T11:08:35.740 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:35.754 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1247,"stamp":"2024-09-16T11:08:35.675227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":456696,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":427766,"kb_avail":565774344,"statfs":{"total":579820584960,"available":579352928256,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":438033062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004157"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":675,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:21.381806+0000","last_change":"2024-09-16T11:08:21.382019+0000","last_active":"2024-09-16T11:08:21.381806+0000","last_peered":"2024-09-16T11:08:21.381806+0000","last_clean":"2024-09-16T11:08:21.381806+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:08:21.381806+0000","last_undegraded":"2024-09-16T11:08:21.381806+0000","last_fullsized":"2024-09-16T11:08:21.381806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:21.381761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:21.381761+0000","last_clean_scrub_stamp":"2024-09-16T11:08:21.381761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:50.028275+0000","scrub_duration":0.0032530609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.443371+0000","last_change":"2024-09-16T11:08:24.443557+0000","last_active":"2024-09-16T11:08:24.443371+0000","last_peered":"2024-09-16T11:08:24.443371+0000","last_clean":"2024-09-16T11:08:24.443371+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:08:24.443371+0000","last_undegraded":"2024-09-16T11:08:24.443371+0000","last_fullsized":"2024-09-16T11:08:24.443371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.443323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.443323+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.443323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:33:35.455955+0000","scrub_duration":0.0031771529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.286459+0000","last_change":"2024-09-16T11:08:24.286739+0000","last_active":"2024-09-16T11:08:24.286459+0000","last_peered":"2024-09-16T11:08:24.286459+0000","last_clean":"2024-09-16T11:08:24.286459+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:08:24.286459+0000","last_undegraded":"2024-09-16T11:08:24.286459+0000","last_fullsized":"2024-09-16T11:08:24.286459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.286405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.286405+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.286405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:48.581623+0000","scrub_duration":0.0026438450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":643,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.660242+0000","last_change":"2024-09-16T11:08:15.660539+0000","last_active":"2024-09-16T11:08:15.660242+0000","last_peered":"2024-09-16T11:08:15.660242+0000","last_clean":"2024-09-16T11:08:15.660242+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:08:15.660242+0000","last_undegraded":"2024-09-16T11:08:15.660242+0000","last_fullsized":"2024-09-16T11:08:15.660242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.660195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.660195+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.660195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:46:16.821519+0000","scrub_duration":0.0027333980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":662,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:16.103482+0000","last_change":"2024-09-16T11:08:16.103670+0000","last_active":"2024-09-16T11:08:16.103482+0000","last_peered":"2024-09-16T11:08:16.103482+0000","last_clean":"2024-09-16T11:08:16.103482+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:08:16.103482+0000","last_undegraded":"2024-09-16T11:08:16.103482+0000","last_fullsized":"2024-09-16T11:08:16.103482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:16.103435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:16.103435+0000","last_clean_scrub_stamp":"2024-09-16T11:08:16.103435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:50.172698+0000","scrub_duration":0.0026738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":645,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:28.416865+0000","last_change":"2024-09-16T11:08:28.416865+0000","last_active":"2024-09-16T11:08:28.416865+0000","last_peered":"2024-09-16T11:08:28.416865+0000","last_clean":"2024-09-16T11:08:28.416865+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:08:28.416865+0000","last_undegraded":"2024-09-16T11:08:28.416865+0000","last_fullsized":"2024-09-16T11:08:28.416865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:28.416610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:28.416610+0000","last_clean_scrub_stamp":"2024-09-16T11:08:28.416610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:31.908883+0000","scrub_duration":0.0030361390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.016188+0000","last_change":"2024-09-16T11:08:19.016188+0000","last_active":"2024-09-16T11:08:19.016188+0000","last_peered":"2024-09-16T11:08:19.016188+0000","last_clean":"2024-09-16T11:08:19.016188+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:08:19.016188+0000","last_undegraded":"2024-09-16T11:08:19.016188+0000","last_fullsized":"2024-09-16T11:08:19.016188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:19.015883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:19.015883+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.015883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:03:28.004509+0000","scrub_duration":0.0033260809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":659,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:25.287713+0000","last_change":"2024-09-16T11:08:25.287713+0000","last_active":"2024-09-16T11:08:25.287713+0000","last_peered":"2024-09-16T11:08:25.287713+0000","last_clean":"2024-09-16T11:08:25.287713+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:08:25.287713+0000","last_undegraded":"2024-09-16T11:08:25.287713+0000","last_fullsized":"2024-09-16T11:08:25.287713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:25.287460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:25.287460+0000","last_clean_scrub_stamp":"2024-09-16T11:08:25.287460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:49.361095+0000","scrub_duration":0.0028583419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":645,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.560300+0000","last_change":"2024-09-16T11:08:18.560300+0000","last_active":"2024-09-16T11:08:18.560300+0000","last_peered":"2024-09-16T11:08:18.560300+0000","last_clean":"2024-09-16T11:08:18.560300+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:08:18.560300+0000","last_undegraded":"2024-09-16T11:08:18.560300+0000","last_fullsized":"2024-09-16T11:08:18.560300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.560067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.560067+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.560067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:24.099562+0000","scrub_duration":0.0031221669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:20.359828+0000","last_change":"2024-09-16T11:08:20.359828+0000","last_active":"2024-09-16T11:08:20.359828+0000","last_peered":"2024-09-16T11:08:20.359828+0000","last_clean":"2024-09-16T11:08:20.359828+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:08:20.359828+0000","last_undegraded":"2024-09-16T11:08:20.359828+0000","last_fullsized":"2024-09-16T11:08:20.359828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:20.359590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:20.359590+0000","last_clean_scrub_stamp":"2024-09-16T11:08:20.359590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:29.770557+0000","scrub_duration":0.0031497180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.132947+0000","last_change":"2024-09-16T11:08:15.133169+0000","last_active":"2024-09-16T11:08:15.132947+0000","last_peered":"2024-09-16T11:08:15.132947+0000","last_clean":"2024-09-16T11:08:15.132947+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:08:15.132947+0000","last_undegraded":"2024-09-16T11:08:15.132947+0000","last_fullsized":"2024-09-16T11:08:15.132947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.132900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.132900+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.132900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:19.575767+0000","scrub_duration":0.0030294499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":659,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.303029+0000","last_change":"2024-09-16T11:08:23.303229+0000","last_active":"2024-09-16T11:08:23.303029+0000","last_peered":"2024-09-16T11:08:23.303029+0000","last_clean":"2024-09-16T11:08:23.303029+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:08:23.303029+0000","last_undegraded":"2024-09-16T11:08:23.303029+0000","last_fullsized":"2024-09-16T11:08:23.303029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.302985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.302985+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.302985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:28:16.088291+0000","scrub_duration":0.0032583550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":644,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.521410+0000","last_change":"2024-09-16T11:08:23.521606+0000","last_active":"2024-09-16T11:08:23.521410+0000","last_peered":"2024-09-16T11:08:23.521410+0000","last_clean":"2024-09-16T11:08:23.521410+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:08:23.521410+0000","last_undegraded":"2024-09-16T11:08:23.521410+0000","last_fullsized":"2024-09-16T11:08:23.521410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.521367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.521367+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.521367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.109691+0000","scrub_duration":0.002795957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.358815+0000","last_change":"2024-09-16T11:08:18.359035+0000","last_active":"2024-09-16T11:08:18.358815+0000","last_peered":"2024-09-16T11:08:18.358815+0000","last_clean":"2024-09-16T11:08:18.358815+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:08:18.358815+0000","last_undegraded":"2024-09-16T11:08:18.358815+0000","last_fullsized":"2024-09-16T11:08:18.358815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.358772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.358772+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.358772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:01.218317+0000","scrub_duration":0.0028433379999999999,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":675,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.356899+0000","last_change":"2024-09-16T11:08:23.357092+0000","last_active":"2024-09-16T11:08:23.356899+0000","last_peered":"2024-09-16T11:08:23.356899+0000","last_clean":"2024-09-16T11:08:23.356899+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:08:23.356899+0000","last_undegraded":"2024-09-16T11:08:23.356899+0000","last_fullsized":"2024-09-16T11:08:23.356899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.356854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.356854+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.356854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:09.556218+0000","scrub_duration":0.0031843100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":670,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.400707+0000","last_change":"2024-09-16T11:08:15.400789+0000","last_active":"2024-09-16T11:08:15.400707+0000","last_peered":"2024-09-16T11:08:15.400707+0000","last_clean":"2024-09-16T11:08:15.400707+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:08:15.400707+0000","last_undegraded":"2024-09-16T11:08:15.400707+0000","last_fullsized":"2024-09-16T11:08:15.400707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.400690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.400690+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.400690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:43.024892+0000","scrub_duration":0.0021878269999999998,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":566,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:30.318321+0000","last_change":"2024-09-16T11:08:30.318638+0000","last_active":"2024-09-16T11:08:30.318321+0000","last_peered":"2024-09-16T11:08:30.318321+0000","last_clean":"2024-09-16T11:08:30.318321+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:08:30.318321+0000","last_undegraded":"2024-09-16T11:08:30.318321+0000","last_fullsized":"2024-09-16T11:08:30.318321+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:30.318231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:30.318231+0000","last_clean_scrub_stamp":"2024-09-16T11:08:30.318231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:19.406941+0000","scrub_duration":0.003193109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:17.311053+0000","last_change":"2024-09-16T11:08:17.311238+0000","last_active":"2024-09-16T11:08:17.311053+0000","last_peered":"2024-09-16T11:08:17.311053+0000","last_clean":"2024-09-16T11:08:17.311053+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:08:17.311053+0000","last_undegraded":"2024-09-16T11:08:17.311053+0000","last_fullsized":"2024-09-16T11:08:17.311053+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:17.310997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:17.310997+0000","last_clean_scrub_stamp":"2024-09-16T11:08:17.310997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:22:27.030828+0000","scrub_duration":0.003208743,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":662,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.010681+0000","last_change":"2024-09-16T11:08:18.010953+0000","last_active":"2024-09-16T11:08:18.010681+0000","last_peered":"2024-09-16T11:08:18.010681+0000","last_clean":"2024-09-16T11:08:18.010681+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:08:18.010681+0000","last_undegraded":"2024-09-16T11:08:18.010681+0000","last_fullsized":"2024-09-16T11:08:18.010681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.010594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.010594+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.010594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:11.319721+0000","scrub_duration":0.0033088079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":670,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:14.426657+0000","last_change":"2024-09-16T11:08:14.426804+0000","last_active":"2024-09-16T11:08:14.426657+0000","last_peered":"2024-09-16T11:08:14.426657+0000","last_clean":"2024-09-16T11:08:14.426657+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:08:14.426657+0000","last_undegraded":"2024-09-16T11:08:14.426657+0000","last_fullsized":"2024-09-16T11:08:14.426657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:14.426625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:14.426625+0000","last_clean_scrub_stamp":"2024-09-16T11:08:14.426625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:08.355682+0000","scrub_duration":0.0019731089999999998,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"694'3","reported_seq":690,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.450319+0000","last_change":"2024-09-16T11:08:18.450613+0000","last_active":"2024-09-16T11:08:18.450319+0000","last_peered":"2024-09-16T11:08:18.450319+0000","last_clean":"2024-09-16T11:08:18.450319+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:08:18.450319+0000","last_undegraded":"2024-09-16T11:08:18.450319+0000","last_fullsized":"2024-09-16T11:08:18.450319+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'3","last_scrub_stamp":"2024-09-16T11:08:18.450207+0000","last_deep_scrub":"694'3","last_deep_scrub_stamp":"2024-09-16T11:08:18.450207+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.450207+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:22:57.269158+0000","scrub_duration":0.0065405519999999998,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":644,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.520503+0000","last_change":"2024-09-16T11:08:24.520698+0000","last_active":"2024-09-16T11:08:24.520503+0000","last_peered":"2024-09-16T11:08:24.520503+0000","last_clean":"2024-09-16T11:08:24.520503+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:08:24.520503+0000","last_undegraded":"2024-09-16T11:08:24.520503+0000","last_fullsized":"2024-09-16T11:08:24.520503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.520459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.520459+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.520459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:42.583634+0000","scrub_duration":0.0030661989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.594239+0000","last_change":"2024-09-16T11:08:15.594430+0000","last_active":"2024-09-16T11:08:15.594239+0000","last_peered":"2024-09-16T11:08:15.594239+0000","last_clean":"2024-09-16T11:08:15.594239+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:08:15.594239+0000","last_undegraded":"2024-09-16T11:08:15.594239+0000","last_fullsized":"2024-09-16T11:08:15.594239+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.594195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.594195+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.594195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:38:36.097879+0000","scrub_duration":0.0029188249999999999,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:28.361813+0000","last_change":"2024-09-16T11:08:28.361996+0000","last_active":"2024-09-16T11:08:28.361813+0000","last_peered":"2024-09-16T11:08:28.361813+0000","last_clean":"2024-09-16T11:08:28.361813+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:08:28.361813+0000","last_undegraded":"2024-09-16T11:08:28.361813+0000","last_fullsized":"2024-09-16T11:08:28.361813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:28.361769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:28.361769+0000","last_clean_scrub_stamp":"2024-09-16T11:08:28.361769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:39.996454+0000","scrub_duration":0.0030839579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:16.353971+0000","last_change":"2024-09-16T11:08:16.354180+0000","last_active":"2024-09-16T11:08:16.353971+0000","last_peered":"2024-09-16T11:08:16.353971+0000","last_clean":"2024-09-16T11:08:16.353971+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:08:16.353971+0000","last_undegraded":"2024-09-16T11:08:16.353971+0000","last_fullsized":"2024-09-16T11:08:16.353971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:16.353925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:16.353925+0000","last_clean_scrub_stamp":"2024-09-16T11:08:16.353925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:45.287512+0000","scrub_duration":0.0031890730000000002,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:33.357547+0000","last_change":"2024-09-16T11:08:33.357735+0000","last_active":"2024-09-16T11:08:33.357547+0000","last_peered":"2024-09-16T11:08:33.357547+0000","last_clean":"2024-09-16T11:08:33.357547+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:08:33.357547+0000","last_undegraded":"2024-09-16T11:08:33.357547+0000","last_fullsized":"2024-09-16T11:08:33.357547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:33.357498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:33.357498+0000","last_clean_scrub_stamp":"2024-09-16T11:08:33.357498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:57.785985+0000","scrub_duration":0.0029415729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"694'2","reported_seq":689,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.470238+0000","last_change":"2024-09-16T11:08:19.470454+0000","last_active":"2024-09-16T11:08:19.470238+0000","last_peered":"2024-09-16T11:08:19.470238+0000","last_clean":"2024-09-16T11:08:19.470238+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:08:19.470238+0000","last_undegraded":"2024-09-16T11:08:19.470238+0000","last_fullsized":"2024-09-16T11:08:19.470238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'2","last_scrub_stamp":"2024-09-16T11:08:19.470184+0000","last_deep_scrub":"694'2","last_deep_scrub_stamp":"2024-09-16T11:08:19.470184+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.470184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:38.599011+0000","scrub_duration":0.005601969,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"694'119","reported_seq":811,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:14.372583+0000","last_change":"2024-09-16T11:08:14.372827+0000","last_active":"2024-09-16T11:08:14.372583+0000","last_peered":"2024-09-16T11:08:14.372583+0000","last_clean":"2024-09-16T11:08:14.372583+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:08:14.372583+0000","last_undegraded":"2024-09-16T11:08:14.372583+0000","last_fullsized":"2024-09-16T11:08:14.372583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'119","last_scrub_stamp":"2024-09-16T11:08:14.372527+0000","last_deep_scrub":"694'119","last_deep_scrub_stamp":"2024-09-16T11:08:14.372527+0000","last_clean_scrub_stamp":"2024-09-16T11:08:14.372527+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:00.178727+0000","scrub_duration":0.0080878519999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":649,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:27.409518+0000","last_change":"2024-09-16T11:08:27.409795+0000","last_active":"2024-09-16T11:08:27.409518+0000","last_peered":"2024-09-16T11:08:27.409518+0000","last_clean":"2024-09-16T11:08:27.409518+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:08:27.409518+0000","last_undegraded":"2024-09-16T11:08:27.409518+0000","last_fullsized":"2024-09-16T11:08:27.409518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:27.409470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:27.409470+0000","last_clean_scrub_stamp":"2024-09-16T11:08:27.409470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:33:02.455410+0000","scrub_duration":0.0033275829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":649,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:30.422625+0000","last_change":"2024-09-16T11:08:30.422897+0000","last_active":"2024-09-16T11:08:30.422625+0000","last_peered":"2024-09-16T11:08:30.422625+0000","last_clean":"2024-09-16T11:08:30.422625+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:08:30.422625+0000","last_undegraded":"2024-09-16T11:08:30.422625+0000","last_fullsized":"2024-09-16T11:08:30.422625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:30.422535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:30.422535+0000","last_clean_scrub_stamp":"2024-09-16T11:08:30.422535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:34:00.009557+0000","scrub_duration":0.003178097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:27.340585+0000","last_change":"2024-09-16T11:08:27.340770+0000","last_active":"2024-09-16T11:08:27.340585+0000","last_peered":"2024-09-16T11:08:27.340585+0000","last_clean":"2024-09-16T11:08:27.340585+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:08:27.340585+0000","last_undegraded":"2024-09-16T11:08:27.340585+0000","last_fullsized":"2024-09-16T11:08:27.340585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:27.340541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:27.340541+0000","last_clean_scrub_stamp":"2024-09-16T11:08:27.340541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:45.840149+0000","scrub_duration":0.0029804530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":632,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.958332+0000","last_change":"2024-09-16T10:51:10.890050+0000","last_active":"2024-09-16T11:07:01.958332+0000","last_peered":"2024-09-16T11:07:01.958332+0000","last_clean":"2024-09-16T11:07:01.958332+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:07:01.958332+0000","last_undegraded":"2024-09-16T11:07:01.958332+0000","last_fullsized":"2024-09-16T11:07:01.958332+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":661,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:25.827788+0000","last_change":"2024-09-16T11:08:25.828076+0000","last_active":"2024-09-16T11:08:25.827788+0000","last_peered":"2024-09-16T11:08:25.827788+0000","last_clean":"2024-09-16T11:08:25.827788+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:08:25.827788+0000","last_undegraded":"2024-09-16T11:08:25.827788+0000","last_fullsized":"2024-09-16T11:08:25.827788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:25.827692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:25.827692+0000","last_clean_scrub_stamp":"2024-09-16T11:08:25.827692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:54.825423+0000","scrub_duration":0.0035462250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":551,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:17.674720+0000","last_change":"2024-09-16T11:08:17.674994+0000","last_active":"2024-09-16T11:08:17.674720+0000","last_peered":"2024-09-16T11:08:17.674720+0000","last_clean":"2024-09-16T11:08:17.674720+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:08:17.674720+0000","last_undegraded":"2024-09-16T11:08:17.674720+0000","last_fullsized":"2024-09-16T11:08:17.674720+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:17.674662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:17.674662+0000","last_clean_scrub_stamp":"2024-09-16T11:08:17.674662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:16.419463+0000","scrub_duration":0.0032044130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":636,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:01.233083+0000","last_change":"2024-09-16T10:51:10.773626+0000","last_active":"2024-09-16T11:07:01.233083+0000","last_peered":"2024-09-16T11:07:01.233083+0000","last_clean":"2024-09-16T11:07:01.233083+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:07:01.233083+0000","last_undegraded":"2024-09-16T11:07:01.233083+0000","last_fullsized":"2024-09-16T11:07:01.233083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":643,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:13.636514+0000","last_change":"2024-09-16T11:08:13.636643+0000","last_active":"2024-09-16T11:08:13.636514+0000","last_peered":"2024-09-16T11:08:13.636514+0000","last_clean":"2024-09-16T11:08:13.636514+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:08:13.636514+0000","last_undegraded":"2024-09-16T11:08:13.636514+0000","last_fullsized":"2024-09-16T11:08:13.636514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:13.636480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:13.636480+0000","last_clean_scrub_stamp":"2024-09-16T11:08:13.636480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:26.327719+0000","scrub_duration":0.0030393299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":637,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:54.607397+0000","last_change":"2024-09-16T10:51:10.830383+0000","last_active":"2024-09-16T11:07:54.607397+0000","last_peered":"2024-09-16T11:07:54.607397+0000","last_clean":"2024-09-16T11:07:54.607397+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:07:54.607397+0000","last_undegraded":"2024-09-16T11:07:54.607397+0000","last_fullsized":"2024-09-16T11:07:54.607397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":634,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:07:14.584110+0000","last_change":"2024-09-16T10:51:10.849321+0000","last_active":"2024-09-16T11:07:14.584110+0000","last_peered":"2024-09-16T11:07:14.584110+0000","last_clean":"2024-09-16T11:07:14.584110+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:07:14.584110+0000","last_undegraded":"2024-09-16T11:07:14.584110+0000","last_fullsized":"2024-09-16T11:07:14.584110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:51:09.717022+0000","last_clean_scrub_stamp":"2024-09-16T10:51:09.717022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"694'8","reported_seq":673,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.366157+0000","last_change":"2024-09-16T11:08:19.366353+0000","last_active":"2024-09-16T11:08:19.366157+0000","last_peered":"2024-09-16T11:08:19.366157+0000","last_clean":"2024-09-16T11:08:19.366157+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:08:19.366157+0000","last_undegraded":"2024-09-16T11:08:19.366157+0000","last_fullsized":"2024-09-16T11:08:19.366157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'8","last_scrub_stamp":"2024-09-16T11:08:19.366108+0000","last_deep_scrub":"694'8","last_deep_scrub_stamp":"2024-09-16T11:08:19.366108+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.366108+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:21.199912+0000","scrub_duration":0.0063593720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":676,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:20.422945+0000","last_change":"2024-09-16T11:08:20.423148+0000","last_active":"2024-09-16T11:08:20.422945+0000","last_peered":"2024-09-16T11:08:20.422945+0000","last_clean":"2024-09-16T11:08:20.422945+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:08:20.422945+0000","last_undegraded":"2024-09-16T11:08:20.422945+0000","last_fullsized":"2024-09-16T11:08:20.422945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:20.422897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:20.422897+0000","last_clean_scrub_stamp":"2024-09-16T11:08:20.422897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:28.150274+0000","scrub_duration":0.0029436589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.322629+0000","last_change":"2024-09-16T11:08:19.322913+0000","last_active":"2024-09-16T11:08:19.322629+0000","last_peered":"2024-09-16T11:08:19.322629+0000","last_clean":"2024-09-16T11:08:19.322629+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:08:19.322629+0000","last_undegraded":"2024-09-16T11:08:19.322629+0000","last_fullsized":"2024-09-16T11:08:19.322629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:19.322543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:19.322543+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.322543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:31.817285+0000","scrub_duration":0.0033373700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80044,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291796,"statfs":{"total":96636764160,"available":96554799104,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.74199999999999999,"15min":0.746},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.96799999999999997,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.75800000000000001},"min":{"1min":0.60299999999999998,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.95499999999999996,"5min":1.123,"15min":1.984},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79100000000000004,"15min":0.80800000000000005},"min":{"1min":0.65800000000000003,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.3},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.87,"15min":0.88300000000000001},"min":{"1min":0.74399999999999999,"5min":0.56999999999999995,"15min":0.53000000000000003},"max":{"1min":1.1459999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.84299999999999997}]},{"osd":2,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.77300000000000002,"15min":0.78300000000000003},"min":{"1min":0.68100000000000005,"5min":0.54200000000000004,"15min":0.45400000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.1779999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81599999999999995,"15min":0.82599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94599999999999995,"5min":1.0620000000000001,"15min":1.22},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67600000000000005,"15min":0.73199999999999998},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":1.075,"5min":1.0780000000000001,"15min":1.385},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.82699999999999996,"15min":0.78900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.365},"last":1.006}]},{"osd":4,"last update":"Mon Sep 16 11:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.72299999999999998,"15min":0.71199999999999997},"min":{"1min":0.58999999999999997,"5min":0.53800000000000003,"15min":0.39400000000000002},"max":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.95399999999999996,"15min":0.92900000000000005},"min":{"1min":0.83399999999999996,"5min":0.72799999999999998,"15min":0.499},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.93899999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640481,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80036,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291804,"statfs":{"total":96636764160,"available":96554807296,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78100000000000003,"15min":0.82899999999999996},"min":{"1min":0.59499999999999997,"5min":0.53100000000000003,"15min":0.373},"max":{"1min":0.92300000000000004,"5min":1.169,"15min":5.351},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.78400000000000003},"min":{"1min":0.65600000000000003,"5min":0.51900000000000002,"15min":0.33900000000000002},"max":{"1min":0.97399999999999998,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.85599999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.89300000000000002,"15min":0.91800000000000004},"min":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.0149999999999999,"5min":1.268,"15min":10.134},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.90600000000000003,"15min":0.91200000000000003},"min":{"1min":0.76200000000000001,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.044,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.95999999999999996}]},{"osd":2,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82299999999999995,"15min":0.89200000000000002},"min":{"1min":0.67300000000000004,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.94399999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.68500000000000005,"5min":0.623,"15min":0.46899999999999997},"max":{"1min":1.014,"5min":1.28,"15min":1.3859999999999999},"last":0.91400000000000003}]},{"osd":3,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.73599999999999999},"min":{"1min":0.52500000000000002,"5min":0.498,"15min":0.41899999999999998},"max":{"1min":0.84699999999999998,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.77700000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.44500000000000001},"max":{"1min":1.0609999999999999,"5min":1.268,"15min":5.0579999999999998},"last":0.88}]},{"osd":5,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.76600000000000001},"min":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.153,"15min":3.7770000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.83199999999999996},"min":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.42099999999999999},"max":{"1min":0.89900000000000002,"5min":1.111,"15min":8.5559999999999992},"last":0.71199999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640481,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76460,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295380,"statfs":{"total":96636764160,"available":96558469120,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.78200000000000003},"min":{"1min":0.58899999999999997,"5min":0.52400000000000002,"15min":0.37},"max":{"1min":0.97799999999999998,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.77400000000000002,"15min":0.78200000000000003},"min":{"1min":0.621,"5min":0.53800000000000003,"15min":0.35599999999999998},"max":{"1min":0.93400000000000005,"5min":1.1000000000000001,"15min":1.609},"last":0.64200000000000002}]},{"osd":1,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.871,"15min":0.84599999999999997},"min":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.436},"max":{"1min":1.01,"5min":1.4430000000000001,"15min":1.821},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.85499999999999998,"5min":0.64500000000000002,"15min":0.46899999999999997},"max":{"1min":1.101,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.75900000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.67700000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98899999999999999,"5min":1.2290000000000001,"15min":10.129},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.85299999999999998,"15min":0.876},"min":{"1min":0.65400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.006,"5min":2.3559999999999999,"15min":10.449999999999999},"last":0.84199999999999997}]},{"osd":4,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.66500000000000004,"15min":0.67000000000000004},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.999,"15min":3.4710000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.745,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.47999999999999998,"15min":0.33500000000000002},"max":{"1min":0.98399999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Sep 16 11:07:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.749,"15min":0.77000000000000002},"min":{"1min":0.56200000000000006,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":1.179,"5min":1.1870000000000001,"15min":1.353},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.81200000000000006,"15min":0.84199999999999997},"min":{"1min":0.76000000000000001,"5min":0.52200000000000002,"15min":0.39600000000000002},"max":{"1min":0.92300000000000004,"5min":1.014,"15min":2.625},"last":0.89700000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640481,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71844,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299996,"statfs":{"total":96636764160,"available":96563195904,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.70599999999999996,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70599999999999996,"15min":0.70899999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.39500000000000002},"max":{"1min":0.85799999999999998,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.66200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.754},"min":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.88400000000000001,"5min":1.155,"15min":1.155},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.60099999999999998},"max":{"1min":0.99299999999999999,"5min":1.341,"15min":1.774},"last":1.0409999999999999}]},{"osd":4,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86699999999999999,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.59499999999999997},"max":{"1min":1.0600000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89000000000000001,"15min":0.876},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.59899999999999998},"max":{"1min":1.0249999999999999,"5min":1.101,"15min":1.7090000000000001},"last":1.0029999999999999}]},{"osd":5,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88800000000000001,"15min":0.88},"min":{"1min":0.70299999999999996,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.0640000000000001,"5min":1.397,"15min":3.7989999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.94799999999999995},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.57799999999999996},"max":{"1min":0.96199999999999997,"5min":1.1950000000000001,"15min":3.2799999999999998},"last":0.97399999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640482,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67756,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304084,"statfs":{"total":96636764160,"available":96567382016,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.60299999999999998},"min":{"1min":0.46899999999999997,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72999999999999998,"15min":0.72699999999999998},"min":{"1min":0.59499999999999997,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.85599999999999998,"5min":1.04,"15min":1.845},"last":0.64000000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64600000000000002,"15min":0.66400000000000003},"min":{"1min":0.51900000000000002,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":1.034,"5min":1.034,"15min":4.9260000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.89700000000000002,"15min":2.8370000000000002},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.879,"15min":0.89200000000000002},"min":{"1min":0.76400000000000001,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.95799999999999996,"15min":0.96799999999999997},"min":{"1min":0.746,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.208,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.86199999999999999}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.89700000000000002},"min":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.93899999999999995,"5min":1.2729999999999999,"15min":2.802},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.94899999999999995,"15min":0.94199999999999995},"min":{"1min":0.76700000000000002,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.206,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86499999999999999,"15min":0.90800000000000003},"min":{"1min":0.67900000000000005,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.90200000000000002,"15min":0.90500000000000003},"min":{"1min":0.71299999999999997,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.159,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.84399999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80556,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291284,"statfs":{"total":96636764160,"available":96554274816,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.221},"max":{"1min":0.78800000000000003,"5min":1.014,"15min":2.6469999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.63200000000000001},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.247},"max":{"1min":1.079,"5min":1.079,"15min":2.3479999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71499999999999997,"5min":1.0289999999999999,"15min":1.327},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.83999999999999997,"5min":1.228,"15min":4.1319999999999997},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.85799999999999998,"15min":0.90300000000000002},"min":{"1min":0.69399999999999995,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.079,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.70699999999999996,"5min":0.38400000000000001,"15min":0.33200000000000002},"max":{"1min":1.175,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.88500000000000001,"15min":0.91700000000000004},"min":{"1min":0.73099999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.008,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.86499999999999999,"15min":0.89700000000000002},"min":{"1min":0.74299999999999999,"5min":0.46999999999999997,"15min":0.36699999999999999},"max":{"1min":1.1020000000000001,"5min":1.302,"15min":3.1440000000000001},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Sep 16 11:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.90400000000000003,"15min":0.89200000000000002},"min":{"1min":0.69399999999999995,"5min":0.49399999999999999,"15min":0.45200000000000001},"max":{"1min":1.1479999999999999,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88,"15min":0.94499999999999995},"min":{"1min":0.69399999999999995,"5min":0.435,"15min":0.435},"max":{"1min":1.0409999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.89400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:35.761 INFO:tasks.ceph:pgid 19.1a last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:35.761 INFO:tasks.ceph:pgid 19.1d last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:35.761 INFO:tasks.ceph:pgid 19.1f last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:35.762 INFO:tasks.ceph:pgid 19.0 last_scrub_stamp 2024-09-16T10:51:09.717022+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=51, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:08:35.762 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:08:55.763 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T11:08:55.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 10 monclient: get_monmap_and_config 2024-09-16T11:08:55.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 10 monclient: build_initial_monmap 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 10 monclient: monmap: 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:55.876177+0000 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:55.876177+0000 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:55.878 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:55.879 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 1 Processor -- start 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 1 -- start start 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.872+0000 7f6d3a661700 10 monclient: init 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d3409e8e0 0x7f6d3409ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-c con 0x7f6d3409e8e0 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d3409d6a0 addr v1:172.21.15.5:6789/0 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d3409dfd0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d34070e40 con 0x7f6d3409d6a0 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d341344e0 con 0x7f6d3409dfd0 2024-09-16T11:08:55.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d340a01d0 con 0x7f6d3409e8e0 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient(hunting): authenticate will time out at 1994.452489s 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6d3409d6a0 0x7f6d341336c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47272/0 (socket says 172.21.15.5:47272) 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 1 -- 172.21.15.5:0/3136593808 learned_addr learned my addr 172.21.15.5:0/3136593808 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 1 --2- 172.21.15.5:0/3136593808 >> v2:172.21.15.143:3300/0 conn(0x7f6d3409e8e0 0x7f6d3409ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): get_auth_request con 0x7f6d3409e8e0 auth_method 0 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256044178 0 0) 0x7f6d34070e40 con 0x7f6d3409d6a0 2024-09-16T11:08:55.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _negotiate 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): my global_id is 74787 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d200035d0 con 0x7f6d3409d6a0 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183205697 0 0) 0x7f6d341344e0 con 0x7f6d3409dfd0 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _negotiate 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): my global_id is 94471 2024-09-16T11:08:55.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d34070e40 con 0x7f6d3409dfd0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360463131 0 0) 0x7f6d200035d0 con 0x7f6d3409d6a0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 >> v2:172.21.15.143:3300/0 conn(0x7f6d3409e8e0 msgr2=0x7f6d3409ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 --2- 172.21.15.5:0/3136593808 >> v2:172.21.15.143:3300/0 conn(0x7f6d3409e8e0 0x7f6d3409ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 >> v1:172.21.15.143:6789/0 conn(0x7f6d3409dfd0 legacy=0x7f6d34133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _finish_hunting 0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 monclient: found mon.noname-a-legacy 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.5:6789/0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3409fef0 con 0x7f6d3409d6a0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _finish_auth 0 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 5 monclient: authenticate success, global_id 74787 2024-09-16T11:08:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3038580900 0 0) 0x7f6d28001670 con 0x7f6d3409d6a0 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: dump: 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] but i'm connected to v1:172.21.15.5:6789/0, reconnecting 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 >> v1:172.21.15.5:6789/0 conn(0x7f6d3409d6a0 legacy=0x7f6d341336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d20003bb0 0x7f6d20003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): picked mon.b con 0x7f6d20003bb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d20004870 0x7f6d20004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): picked mon.c con 0x7f6d20004870 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 0x7f6d2000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): picked mon.a con 0x7f6d200052a0 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d340a01d0 con 0x7f6d200052a0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000c200 con 0x7f6d20003bb0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000c3c0 con 0x7f6d20004870 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 0x7f6d2000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d20004870 0x7f6d20004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d20003bb0 0x7f6d20003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): get_auth_request con 0x7f6d200052a0 auth_method 0 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): get_auth_request con 0x7f6d20003bb0 auth_method 0 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): get_auth_request con 0x7f6d20004870 auth_method 0 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient(hunting): handle_auth_done global_id 74787 payload 293 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 -- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d20004870 msgr2=0x7f6d20004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d20004870 0x7f6d20004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 -- 172.21.15.5:0/3136593808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d20003bb0 msgr2=0x7f6d20003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d20003bb0 0x7f6d20003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient: _finish_hunting 0 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 monclient: found mon.a 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 -- 172.21.15.5:0/3136593808 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d200035d0 con 0x7f6d200052a0 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient: _finish_auth 0 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 10 monclient: _check_auth_tickets 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 -- 172.21.15.5:0/3136593808 reap_dead start 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d39e60700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 0x7f6d2000ba30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c002230 tx=0x7f6d200058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d80ad754033614 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:55.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c001f30 con 0x7f6d200052a0 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: dump: 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c00ce80 con 0x7f6d200052a0 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d3a661700 4 set_mon_vals no callback set 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 1 -- 172.21.15.5:0/3136593808 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c003c60 con 0x7f6d200052a0 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.876+0000 7f6d27fff700 10 monclient: dump: 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: shutdown 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 msgr2=0x7f6d2000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 0x7f6d2000ba30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c002230 tx=0x7f6d200058a0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- 172.21.15.5:0/3136593808 shutdown_connections 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 --2- 172.21.15.5:0/3136593808 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d200052a0 0x7f6d2000ba30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- 172.21.15.5:0/3136593808 >> 172.21.15.5:0/3136593808 conn(0x7f6d340a45d0 msgr2=0x7f6d340a09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:55.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- 172.21.15.5:0/3136593808 shutdown_connections 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- 172.21.15.5:0/3136593808 wait complete. 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: build_initial_monmap 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: monmap: 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:55.904 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 librados: starting msgr at 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 librados: starting objecter 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 Processor -- start 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- start start 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 librados: setting wanted keys 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 librados: calling monclient init 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: init 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 0x7f6d341cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-c con 0x7f6d200052a0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 0x7f6d34122060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-a con 0x7f6d341cb130 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341cb570 0x7f6d341225a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): picked mon.noname-b con 0x7f6d341cb570 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3409b0d0 con 0x7f6d341cb130 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d34123d00 con 0x7f6d341cb570 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3409b9e0 con 0x7f6d200052a0 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3a661700 10 monclient(hunting): authenticate will time out at 1994.459686s 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 0x7f6d341cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 0x7f6d34122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 0x7f6d341cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60966/0 (socket says 172.21.15.5:60966) 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 1 -- 172.21.15.5:0/1335670883 learned_addr learned my addr 172.21.15.5:0/1335670883 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 10 monclient(hunting): get_auth_request con 0x7f6d200052a0 auth_method 0 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d3965f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): get_auth_request con 0x7f6d341cb130 auth_method 0 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.880+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d39e60700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341cb570 0x7f6d341225a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d39e60700 10 monclient(hunting): get_auth_request con 0x7f6d341cb570 auth_method 0 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d39e60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d39e60700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d39e60700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 10 monclient(hunting): handle_auth_done global_id 74790 payload 824 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 -- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 msgr2=0x7f6d341cabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 0x7f6d341cabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 -- 172.21.15.5:0/1335670883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341cb570 msgr2=0x7f6d341225a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341cb570 0x7f6d341225a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 10 monclient: _finish_hunting 0 2024-09-16T11:08:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 monclient: found mon.noname-a 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.5:3300/0 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 -- 172.21.15.5:0/1335670883 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3409fab0 con 0x7f6d341cb130 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d38e5e700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 0x7f6d34122060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6d28001eb0 tx=0x7f6d280073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 client.?.objecter ms_handle_connect 0x7f6d341cb130 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28012560 con 0x7f6d341cb130 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: dump: 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: _finish_auth 0 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: _check_auth_tickets 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 5 monclient: authenticate success, global_id 74790 2024-09-16T11:08:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _renew_subs 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d28012700 con 0x7f6d341cb130 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3409b470 con 0x7f6d341cb130 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28012ab0 con 0x7f6d341cb130 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 client.74790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 monclient: dump: 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d277fe700 4 set_mon_vals no callback set 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _renew_subs 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d34126060 con 0x7f6d341cb130 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 1 librados: init done 2024-09-16T11:08:55.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f6d28012c10 con 0x7f6d341cb130 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 --2- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 0x7f6d18039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f6d28047a80 con 0x7f6d341cb130 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 client.74790.objecter ms_dispatch 0x7f6d34000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 3 client.74790.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 3 client.74790.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.5:3300/0 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3409f050 con 0x7f6d341cb130 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 20 client.74790.objecter dump_active .. 0 homeless 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 1 --2- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 0x7f6d18039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:55.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 10 monclient: get_auth_request con 0x7f6d18036f60 auth_method 0 2024-09-16T11:08:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d25ffb700 10 client.74790.objecter ms_handle_connect 0x7f6d18036f60 2024-09-16T11:08:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.884+0000 7f6d3965f700 1 --2- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 0x7f6d18039420 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009d10 tx=0x7f6d30009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.892+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6d280133f0 con 0x7f6d341cb130 2024-09-16T11:08:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.892+0000 7f6d25ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:55.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:55.892+0000 7f6d25ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:56.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.064+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 --> v2:172.21.15.143:6812/1098205034 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d3409ecc0 con 0x7f6d18036f60 2024-09-16T11:08:56.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.072+0000 7f6d25ffb700 1 -- 172.21.15.5:0/1335670883 <== mgr.4102 v2:172.21.15.143:6812/1098205034 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134864 (secure 0 0 0) 0x7f6d30019070 con 0x7f6d18036f60 2024-09-16T11:08:56.079 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-16T11:08:56.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 10 librados: watch_flush enter 2024-09-16T11:08:56.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 10 librados: watch_flush exit 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 20 client.74790.objecter shutdown clearing up homeless session... 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 10 client.74790.objecter successfully canceled tick 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 msgr2=0x7f6d18039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 0x7f6d18039420 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009d10 tx=0x7f6d30009240 comp rx=0 tx=0).stop 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 10 monclient: shutdown 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 msgr2=0x7f6d34122060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 0x7f6d34122060 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6d28001eb0 tx=0x7f6d280073f0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 shutdown_connections 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d341cb570 0x7f6d341225a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> v2:172.21.15.143:6812/1098205034 conn(0x7f6d18036f60 0x7f6d18039420 unknown :-1 s=CLOSED pgs=3930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6d341cb130 0x7f6d34122060 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 --2- 172.21.15.5:0/1335670883 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f6d200052a0 0x7f6d341cabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 >> 172.21.15.5:0/1335670883 conn(0x7f6d340a45d0 msgr2=0x7f6d3406ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 shutdown_connections 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 -- 172.21.15.5:0/1335670883 wait complete. 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.076+0000 7f6d3a661700 1 librados: shutdown 2024-09-16T11:08:56.083 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-16T11:08:56.096 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":1257,"stamp":"2024-09-16T11:08:55.682090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":456696,"kb_used_data":28872,"kb_used_omap":9,"kb_used_meta":427766,"kb_avail":565774344,"statfs":{"total":579820584960,"available":579352928256,"internally_reserved":0,"allocated":29564928,"data_stored":12966212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":438033062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003858"},"pg_stats":[{"pgid":"19.f","version":"0'0","reported_seq":675,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:21.381806+0000","last_change":"2024-09-16T11:08:21.382019+0000","last_active":"2024-09-16T11:08:21.381806+0000","last_peered":"2024-09-16T11:08:21.381806+0000","last_clean":"2024-09-16T11:08:21.381806+0000","last_became_active":"2024-09-16T10:51:10.890973+0000","last_became_peered":"2024-09-16T10:51:10.890973+0000","last_unstale":"2024-09-16T11:08:21.381806+0000","last_undegraded":"2024-09-16T11:08:21.381806+0000","last_fullsized":"2024-09-16T11:08:21.381806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:21.381761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:21.381761+0000","last_clean_scrub_stamp":"2024-09-16T11:08:21.381761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:50.028275+0000","scrub_duration":0.0032530609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.443371+0000","last_change":"2024-09-16T11:08:24.443557+0000","last_active":"2024-09-16T11:08:24.443371+0000","last_peered":"2024-09-16T11:08:24.443371+0000","last_clean":"2024-09-16T11:08:24.443371+0000","last_became_active":"2024-09-16T10:51:10.811613+0000","last_became_peered":"2024-09-16T10:51:10.811613+0000","last_unstale":"2024-09-16T11:08:24.443371+0000","last_undegraded":"2024-09-16T11:08:24.443371+0000","last_fullsized":"2024-09-16T11:08:24.443371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.443323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.443323+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.443323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:33:35.455955+0000","scrub_duration":0.0031771529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.286459+0000","last_change":"2024-09-16T11:08:24.286739+0000","last_active":"2024-09-16T11:08:24.286459+0000","last_peered":"2024-09-16T11:08:24.286459+0000","last_clean":"2024-09-16T11:08:24.286459+0000","last_became_active":"2024-09-16T10:51:10.836135+0000","last_became_peered":"2024-09-16T10:51:10.836135+0000","last_unstale":"2024-09-16T11:08:24.286459+0000","last_undegraded":"2024-09-16T11:08:24.286459+0000","last_fullsized":"2024-09-16T11:08:24.286459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.286405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.286405+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.286405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:40:48.581623+0000","scrub_duration":0.0026438450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":643,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.660242+0000","last_change":"2024-09-16T11:08:15.660539+0000","last_active":"2024-09-16T11:08:15.660242+0000","last_peered":"2024-09-16T11:08:15.660242+0000","last_clean":"2024-09-16T11:08:15.660242+0000","last_became_active":"2024-09-16T10:51:10.853193+0000","last_became_peered":"2024-09-16T10:51:10.853193+0000","last_unstale":"2024-09-16T11:08:15.660242+0000","last_undegraded":"2024-09-16T11:08:15.660242+0000","last_fullsized":"2024-09-16T11:08:15.660242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.660195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.660195+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.660195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:46:16.821519+0000","scrub_duration":0.0027333980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":662,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:16.103482+0000","last_change":"2024-09-16T11:08:16.103670+0000","last_active":"2024-09-16T11:08:16.103482+0000","last_peered":"2024-09-16T11:08:16.103482+0000","last_clean":"2024-09-16T11:08:16.103482+0000","last_became_active":"2024-09-16T10:51:10.888088+0000","last_became_peered":"2024-09-16T10:51:10.888088+0000","last_unstale":"2024-09-16T11:08:16.103482+0000","last_undegraded":"2024-09-16T11:08:16.103482+0000","last_fullsized":"2024-09-16T11:08:16.103482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:16.103435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:16.103435+0000","last_clean_scrub_stamp":"2024-09-16T11:08:16.103435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:08:50.172698+0000","scrub_duration":0.0026738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":645,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:28.416865+0000","last_change":"2024-09-16T11:08:28.416865+0000","last_active":"2024-09-16T11:08:28.416865+0000","last_peered":"2024-09-16T11:08:28.416865+0000","last_clean":"2024-09-16T11:08:28.416865+0000","last_became_active":"2024-09-16T10:51:10.820298+0000","last_became_peered":"2024-09-16T10:51:10.820298+0000","last_unstale":"2024-09-16T11:08:28.416865+0000","last_undegraded":"2024-09-16T11:08:28.416865+0000","last_fullsized":"2024-09-16T11:08:28.416865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:28.416610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:28.416610+0000","last_clean_scrub_stamp":"2024-09-16T11:08:28.416610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:01:31.908883+0000","scrub_duration":0.0030361390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.016188+0000","last_change":"2024-09-16T11:08:19.016188+0000","last_active":"2024-09-16T11:08:19.016188+0000","last_peered":"2024-09-16T11:08:19.016188+0000","last_clean":"2024-09-16T11:08:19.016188+0000","last_became_active":"2024-09-16T10:51:10.901327+0000","last_became_peered":"2024-09-16T10:51:10.901327+0000","last_unstale":"2024-09-16T11:08:19.016188+0000","last_undegraded":"2024-09-16T11:08:19.016188+0000","last_fullsized":"2024-09-16T11:08:19.016188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:19.015883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:19.015883+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.015883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:03:28.004509+0000","scrub_duration":0.0033260809999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":659,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:25.287713+0000","last_change":"2024-09-16T11:08:25.287713+0000","last_active":"2024-09-16T11:08:25.287713+0000","last_peered":"2024-09-16T11:08:25.287713+0000","last_clean":"2024-09-16T11:08:25.287713+0000","last_became_active":"2024-09-16T10:51:10.825664+0000","last_became_peered":"2024-09-16T10:51:10.825664+0000","last_unstale":"2024-09-16T11:08:25.287713+0000","last_undegraded":"2024-09-16T11:08:25.287713+0000","last_fullsized":"2024-09-16T11:08:25.287713+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:25.287460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:25.287460+0000","last_clean_scrub_stamp":"2024-09-16T11:08:25.287460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:59:49.361095+0000","scrub_duration":0.0028583419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":645,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.560300+0000","last_change":"2024-09-16T11:08:18.560300+0000","last_active":"2024-09-16T11:08:18.560300+0000","last_peered":"2024-09-16T11:08:18.560300+0000","last_clean":"2024-09-16T11:08:18.560300+0000","last_became_active":"2024-09-16T10:51:10.837499+0000","last_became_peered":"2024-09-16T10:51:10.837499+0000","last_unstale":"2024-09-16T11:08:18.560300+0000","last_undegraded":"2024-09-16T11:08:18.560300+0000","last_fullsized":"2024-09-16T11:08:18.560300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.560067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.560067+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.560067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:20:24.099562+0000","scrub_duration":0.0031221669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:20.359828+0000","last_change":"2024-09-16T11:08:20.359828+0000","last_active":"2024-09-16T11:08:20.359828+0000","last_peered":"2024-09-16T11:08:20.359828+0000","last_clean":"2024-09-16T11:08:20.359828+0000","last_became_active":"2024-09-16T10:51:10.816056+0000","last_became_peered":"2024-09-16T10:51:10.816056+0000","last_unstale":"2024-09-16T11:08:20.359828+0000","last_undegraded":"2024-09-16T11:08:20.359828+0000","last_fullsized":"2024-09-16T11:08:20.359828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:20.359590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:20.359590+0000","last_clean_scrub_stamp":"2024-09-16T11:08:20.359590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:29.770557+0000","scrub_duration":0.0031497180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.132947+0000","last_change":"2024-09-16T11:08:15.133169+0000","last_active":"2024-09-16T11:08:15.132947+0000","last_peered":"2024-09-16T11:08:15.132947+0000","last_clean":"2024-09-16T11:08:15.132947+0000","last_became_active":"2024-09-16T10:51:10.914806+0000","last_became_peered":"2024-09-16T10:51:10.914806+0000","last_unstale":"2024-09-16T11:08:15.132947+0000","last_undegraded":"2024-09-16T11:08:15.132947+0000","last_fullsized":"2024-09-16T11:08:15.132947+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.132900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.132900+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.132900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:19.575767+0000","scrub_duration":0.0030294499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":659,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.303029+0000","last_change":"2024-09-16T11:08:23.303229+0000","last_active":"2024-09-16T11:08:23.303029+0000","last_peered":"2024-09-16T11:08:23.303029+0000","last_clean":"2024-09-16T11:08:23.303029+0000","last_became_active":"2024-09-16T10:51:10.806952+0000","last_became_peered":"2024-09-16T10:51:10.806952+0000","last_unstale":"2024-09-16T11:08:23.303029+0000","last_undegraded":"2024-09-16T11:08:23.303029+0000","last_fullsized":"2024-09-16T11:08:23.303029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.302985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.302985+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.302985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:28:16.088291+0000","scrub_duration":0.0032583550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":644,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.521410+0000","last_change":"2024-09-16T11:08:23.521606+0000","last_active":"2024-09-16T11:08:23.521410+0000","last_peered":"2024-09-16T11:08:23.521410+0000","last_clean":"2024-09-16T11:08:23.521410+0000","last_became_active":"2024-09-16T10:51:10.888143+0000","last_became_peered":"2024-09-16T10:51:10.888143+0000","last_unstale":"2024-09-16T11:08:23.521410+0000","last_undegraded":"2024-09-16T11:08:23.521410+0000","last_fullsized":"2024-09-16T11:08:23.521410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.521367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.521367+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.521367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:31.109691+0000","scrub_duration":0.002795957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":660,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.358815+0000","last_change":"2024-09-16T11:08:18.359035+0000","last_active":"2024-09-16T11:08:18.358815+0000","last_peered":"2024-09-16T11:08:18.358815+0000","last_clean":"2024-09-16T11:08:18.358815+0000","last_became_active":"2024-09-16T10:50:01.420179+0000","last_became_peered":"2024-09-16T10:50:01.420179+0000","last_unstale":"2024-09-16T11:08:18.358815+0000","last_undegraded":"2024-09-16T11:08:18.358815+0000","last_fullsized":"2024-09-16T11:08:18.358815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.358772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.358772+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.358772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:45:01.218317+0000","scrub_duration":0.0028433379999999999,"objects_trimmed":0,"snaptrim_duration":0.00058392999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":675,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:23.356899+0000","last_change":"2024-09-16T11:08:23.357092+0000","last_active":"2024-09-16T11:08:23.356899+0000","last_peered":"2024-09-16T11:08:23.356899+0000","last_clean":"2024-09-16T11:08:23.356899+0000","last_became_active":"2024-09-16T10:51:10.904798+0000","last_became_peered":"2024-09-16T10:51:10.904798+0000","last_unstale":"2024-09-16T11:08:23.356899+0000","last_undegraded":"2024-09-16T11:08:23.356899+0000","last_fullsized":"2024-09-16T11:08:23.356899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:23.356854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:23.356854+0000","last_clean_scrub_stamp":"2024-09-16T11:08:23.356854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:09.556218+0000","scrub_duration":0.0031843100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":670,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.400707+0000","last_change":"2024-09-16T11:08:15.400789+0000","last_active":"2024-09-16T11:08:15.400707+0000","last_peered":"2024-09-16T11:08:15.400707+0000","last_clean":"2024-09-16T11:08:15.400707+0000","last_became_active":"2024-09-16T10:50:01.406883+0000","last_became_peered":"2024-09-16T10:50:01.406883+0000","last_unstale":"2024-09-16T11:08:15.400707+0000","last_undegraded":"2024-09-16T11:08:15.400707+0000","last_fullsized":"2024-09-16T11:08:15.400707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.400690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.400690+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.400690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:43.024892+0000","scrub_duration":0.0021878269999999998,"objects_trimmed":0,"snaptrim_duration":0.00054742300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":566,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:30.318321+0000","last_change":"2024-09-16T11:08:30.318638+0000","last_active":"2024-09-16T11:08:30.318321+0000","last_peered":"2024-09-16T11:08:30.318321+0000","last_clean":"2024-09-16T11:08:30.318321+0000","last_became_active":"2024-09-16T10:52:56.721891+0000","last_became_peered":"2024-09-16T10:52:56.721891+0000","last_unstale":"2024-09-16T11:08:30.318321+0000","last_undegraded":"2024-09-16T11:08:30.318321+0000","last_fullsized":"2024-09-16T11:08:30.318321+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:30.318231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:30.318231+0000","last_clean_scrub_stamp":"2024-09-16T11:08:30.318231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:19.406941+0000","scrub_duration":0.003193109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":661,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:17.311238+0000","last_change":"2024-09-16T11:08:17.311238+0000","last_active":"2024-09-16T11:08:17.311238+0000","last_peered":"2024-09-16T11:08:17.311238+0000","last_clean":"2024-09-16T11:08:17.311238+0000","last_became_active":"2024-09-16T10:50:01.403259+0000","last_became_peered":"2024-09-16T10:50:01.403259+0000","last_unstale":"2024-09-16T11:08:17.311238+0000","last_undegraded":"2024-09-16T11:08:17.311238+0000","last_fullsized":"2024-09-16T11:08:17.311238+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:17.310997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:17.310997+0000","last_clean_scrub_stamp":"2024-09-16T11:08:17.310997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:22:27.030828+0000","scrub_duration":0.003208743,"objects_trimmed":0,"snaptrim_duration":0.00067442499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":662,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.010681+0000","last_change":"2024-09-16T11:08:18.010953+0000","last_active":"2024-09-16T11:08:18.010681+0000","last_peered":"2024-09-16T11:08:18.010681+0000","last_clean":"2024-09-16T11:08:18.010681+0000","last_became_active":"2024-09-16T10:51:10.906227+0000","last_became_peered":"2024-09-16T10:51:10.906227+0000","last_unstale":"2024-09-16T11:08:18.010681+0000","last_undegraded":"2024-09-16T11:08:18.010681+0000","last_fullsized":"2024-09-16T11:08:18.010681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:18.010594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:18.010594+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.010594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:11.319721+0000","scrub_duration":0.0033088079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":671,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:14.426804+0000","last_change":"2024-09-16T11:08:14.426804+0000","last_active":"2024-09-16T11:08:14.426804+0000","last_peered":"2024-09-16T11:08:14.426804+0000","last_clean":"2024-09-16T11:08:14.426804+0000","last_became_active":"2024-09-16T10:50:01.401499+0000","last_became_peered":"2024-09-16T10:50:01.401499+0000","last_unstale":"2024-09-16T11:08:14.426804+0000","last_undegraded":"2024-09-16T11:08:14.426804+0000","last_fullsized":"2024-09-16T11:08:14.426804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:14.426625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:14.426625+0000","last_clean_scrub_stamp":"2024-09-16T11:08:14.426625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:08.355682+0000","scrub_duration":0.0019731089999999998,"objects_trimmed":0,"snaptrim_duration":0.000724583,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"694'3","reported_seq":691,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:18.450613+0000","last_change":"2024-09-16T11:08:18.450613+0000","last_active":"2024-09-16T11:08:18.450613+0000","last_peered":"2024-09-16T11:08:18.450613+0000","last_clean":"2024-09-16T11:08:18.450613+0000","last_became_active":"2024-09-16T10:50:01.406550+0000","last_became_peered":"2024-09-16T10:50:01.406550+0000","last_unstale":"2024-09-16T11:08:18.450613+0000","last_undegraded":"2024-09-16T11:08:18.450613+0000","last_fullsized":"2024-09-16T11:08:18.450613+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'3","last_scrub_stamp":"2024-09-16T11:08:18.450207+0000","last_deep_scrub":"694'3","last_deep_scrub_stamp":"2024-09-16T11:08:18.450207+0000","last_clean_scrub_stamp":"2024-09-16T11:08:18.450207+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:22:57.269158+0000","scrub_duration":0.0065405519999999998,"objects_trimmed":0,"snaptrim_duration":0.0016754129999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":644,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:24.520503+0000","last_change":"2024-09-16T11:08:24.520698+0000","last_active":"2024-09-16T11:08:24.520503+0000","last_peered":"2024-09-16T11:08:24.520503+0000","last_clean":"2024-09-16T11:08:24.520503+0000","last_became_active":"2024-09-16T10:51:10.897501+0000","last_became_peered":"2024-09-16T10:51:10.897501+0000","last_unstale":"2024-09-16T11:08:24.520503+0000","last_undegraded":"2024-09-16T11:08:24.520503+0000","last_fullsized":"2024-09-16T11:08:24.520503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:24.520459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:24.520459+0000","last_clean_scrub_stamp":"2024-09-16T11:08:24.520459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:42.583634+0000","scrub_duration":0.0030661989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":657,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:15.594430+0000","last_change":"2024-09-16T11:08:15.594430+0000","last_active":"2024-09-16T11:08:15.594430+0000","last_peered":"2024-09-16T11:08:15.594430+0000","last_clean":"2024-09-16T11:08:15.594430+0000","last_became_active":"2024-09-16T10:50:01.402541+0000","last_became_peered":"2024-09-16T10:50:01.402541+0000","last_unstale":"2024-09-16T11:08:15.594430+0000","last_undegraded":"2024-09-16T11:08:15.594430+0000","last_fullsized":"2024-09-16T11:08:15.594430+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:15.594195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:15.594195+0000","last_clean_scrub_stamp":"2024-09-16T11:08:15.594195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:38:36.097879+0000","scrub_duration":0.0029188249999999999,"objects_trimmed":0,"snaptrim_duration":0.000461094,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:28.361813+0000","last_change":"2024-09-16T11:08:28.361996+0000","last_active":"2024-09-16T11:08:28.361813+0000","last_peered":"2024-09-16T11:08:28.361813+0000","last_clean":"2024-09-16T11:08:28.361813+0000","last_became_active":"2024-09-16T10:51:10.776680+0000","last_became_peered":"2024-09-16T10:51:10.776680+0000","last_unstale":"2024-09-16T11:08:28.361813+0000","last_undegraded":"2024-09-16T11:08:28.361813+0000","last_fullsized":"2024-09-16T11:08:28.361813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:28.361769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:28.361769+0000","last_clean_scrub_stamp":"2024-09-16T11:08:28.361769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:39.996454+0000","scrub_duration":0.0030839579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":663,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:16.353971+0000","last_change":"2024-09-16T11:08:16.354180+0000","last_active":"2024-09-16T11:08:16.353971+0000","last_peered":"2024-09-16T11:08:16.353971+0000","last_clean":"2024-09-16T11:08:16.353971+0000","last_became_active":"2024-09-16T10:50:01.403374+0000","last_became_peered":"2024-09-16T10:50:01.403374+0000","last_unstale":"2024-09-16T11:08:16.353971+0000","last_undegraded":"2024-09-16T11:08:16.353971+0000","last_fullsized":"2024-09-16T11:08:16.353971+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:16.353925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:16.353925+0000","last_clean_scrub_stamp":"2024-09-16T11:08:16.353925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:45.287512+0000","scrub_duration":0.0031890730000000002,"objects_trimmed":0,"snaptrim_duration":0.0011005450000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:33.357547+0000","last_change":"2024-09-16T11:08:33.357735+0000","last_active":"2024-09-16T11:08:33.357547+0000","last_peered":"2024-09-16T11:08:33.357547+0000","last_clean":"2024-09-16T11:08:33.357547+0000","last_became_active":"2024-09-16T10:51:10.818058+0000","last_became_peered":"2024-09-16T10:51:10.818058+0000","last_unstale":"2024-09-16T11:08:33.357547+0000","last_undegraded":"2024-09-16T11:08:33.357547+0000","last_fullsized":"2024-09-16T11:08:33.357547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:33.357498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:33.357498+0000","last_clean_scrub_stamp":"2024-09-16T11:08:33.357498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:57.785985+0000","scrub_duration":0.0029415729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"694'2","reported_seq":690,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.470454+0000","last_change":"2024-09-16T11:08:19.470454+0000","last_active":"2024-09-16T11:08:19.470454+0000","last_peered":"2024-09-16T11:08:19.470454+0000","last_clean":"2024-09-16T11:08:19.470454+0000","last_became_active":"2024-09-16T10:50:01.407241+0000","last_became_peered":"2024-09-16T10:50:01.407241+0000","last_unstale":"2024-09-16T11:08:19.470454+0000","last_undegraded":"2024-09-16T11:08:19.470454+0000","last_fullsized":"2024-09-16T11:08:19.470454+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'2","last_scrub_stamp":"2024-09-16T11:08:19.470184+0000","last_deep_scrub":"694'2","last_deep_scrub_stamp":"2024-09-16T11:08:19.470184+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.470184+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:38.599011+0000","scrub_duration":0.005601969,"objects_trimmed":0,"snaptrim_duration":0.00079171799999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"694'119","reported_seq":811,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:14.372583+0000","last_change":"2024-09-16T11:08:14.372827+0000","last_active":"2024-09-16T11:08:14.372583+0000","last_peered":"2024-09-16T11:08:14.372583+0000","last_clean":"2024-09-16T11:08:14.372583+0000","last_became_active":"2024-09-16T10:50:00.397066+0000","last_became_peered":"2024-09-16T10:50:00.397066+0000","last_unstale":"2024-09-16T11:08:14.372583+0000","last_undegraded":"2024-09-16T11:08:14.372583+0000","last_fullsized":"2024-09-16T11:08:14.372583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'119","last_scrub_stamp":"2024-09-16T11:08:14.372527+0000","last_deep_scrub":"694'119","last_deep_scrub_stamp":"2024-09-16T11:08:14.372527+0000","last_clean_scrub_stamp":"2024-09-16T11:08:14.372527+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:00.178727+0000","scrub_duration":0.0080878519999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":649,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:27.409518+0000","last_change":"2024-09-16T11:08:27.409795+0000","last_active":"2024-09-16T11:08:27.409518+0000","last_peered":"2024-09-16T11:08:27.409518+0000","last_clean":"2024-09-16T11:08:27.409518+0000","last_became_active":"2024-09-16T10:51:10.819418+0000","last_became_peered":"2024-09-16T10:51:10.819418+0000","last_unstale":"2024-09-16T11:08:27.409518+0000","last_undegraded":"2024-09-16T11:08:27.409518+0000","last_fullsized":"2024-09-16T11:08:27.409518+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:27.409470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:27.409470+0000","last_clean_scrub_stamp":"2024-09-16T11:08:27.409470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:33:02.455410+0000","scrub_duration":0.0033275829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":649,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:30.422625+0000","last_change":"2024-09-16T11:08:30.422897+0000","last_active":"2024-09-16T11:08:30.422625+0000","last_peered":"2024-09-16T11:08:30.422625+0000","last_clean":"2024-09-16T11:08:30.422625+0000","last_became_active":"2024-09-16T10:51:10.880634+0000","last_became_peered":"2024-09-16T10:51:10.880634+0000","last_unstale":"2024-09-16T11:08:30.422625+0000","last_undegraded":"2024-09-16T11:08:30.422625+0000","last_fullsized":"2024-09-16T11:08:30.422625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:30.422535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:30.422535+0000","last_clean_scrub_stamp":"2024-09-16T11:08:30.422535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:34:00.009557+0000","scrub_duration":0.003178097,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":658,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:27.340585+0000","last_change":"2024-09-16T11:08:27.340770+0000","last_active":"2024-09-16T11:08:27.340585+0000","last_peered":"2024-09-16T11:08:27.340585+0000","last_clean":"2024-09-16T11:08:27.340585+0000","last_became_active":"2024-09-16T10:51:10.821835+0000","last_became_peered":"2024-09-16T10:51:10.821835+0000","last_unstale":"2024-09-16T11:08:27.340585+0000","last_undegraded":"2024-09-16T11:08:27.340585+0000","last_fullsized":"2024-09-16T11:08:27.340585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:27.340541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:27.340541+0000","last_clean_scrub_stamp":"2024-09-16T11:08:27.340541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:45.840149+0000","scrub_duration":0.0029804530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":644,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:37.625566+0000","last_change":"2024-09-16T11:08:37.625566+0000","last_active":"2024-09-16T11:08:37.625566+0000","last_peered":"2024-09-16T11:08:37.625566+0000","last_clean":"2024-09-16T11:08:37.625566+0000","last_became_active":"2024-09-16T10:51:10.888873+0000","last_became_peered":"2024-09-16T10:51:10.888873+0000","last_unstale":"2024-09-16T11:08:37.625566+0000","last_undegraded":"2024-09-16T11:08:37.625566+0000","last_fullsized":"2024-09-16T11:08:37.625566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:37.625324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:37.625324+0000","last_clean_scrub_stamp":"2024-09-16T11:08:37.625324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:40:21.715805+0000","scrub_duration":0.0032285780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":662,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:25.828076+0000","last_change":"2024-09-16T11:08:25.828076+0000","last_active":"2024-09-16T11:08:25.828076+0000","last_peered":"2024-09-16T11:08:25.828076+0000","last_clean":"2024-09-16T11:08:25.828076+0000","last_became_active":"2024-09-16T10:51:10.849819+0000","last_became_peered":"2024-09-16T10:51:10.849819+0000","last_unstale":"2024-09-16T11:08:25.828076+0000","last_undegraded":"2024-09-16T11:08:25.828076+0000","last_fullsized":"2024-09-16T11:08:25.828076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:25.827692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:25.827692+0000","last_clean_scrub_stamp":"2024-09-16T11:08:25.827692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:12:54.825423+0000","scrub_duration":0.0035462250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":551,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:17.674720+0000","last_change":"2024-09-16T11:08:17.674994+0000","last_active":"2024-09-16T11:08:17.674720+0000","last_peered":"2024-09-16T11:08:17.674720+0000","last_clean":"2024-09-16T11:08:17.674720+0000","last_became_active":"2024-09-16T10:52:56.729598+0000","last_became_peered":"2024-09-16T10:52:56.729598+0000","last_unstale":"2024-09-16T11:08:17.674720+0000","last_undegraded":"2024-09-16T11:08:17.674720+0000","last_fullsized":"2024-09-16T11:08:17.674720+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:17.674662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:17.674662+0000","last_clean_scrub_stamp":"2024-09-16T11:08:17.674662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:16.419463+0000","scrub_duration":0.0032044130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:35.330820+0000","last_change":"2024-09-16T11:08:35.330820+0000","last_active":"2024-09-16T11:08:35.330820+0000","last_peered":"2024-09-16T11:08:35.330820+0000","last_clean":"2024-09-16T11:08:35.330820+0000","last_became_active":"2024-09-16T10:51:10.773080+0000","last_became_peered":"2024-09-16T10:51:10.773080+0000","last_unstale":"2024-09-16T11:08:35.330820+0000","last_undegraded":"2024-09-16T11:08:35.330820+0000","last_fullsized":"2024-09-16T11:08:35.330820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:35.330592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:35.330592+0000","last_clean_scrub_stamp":"2024-09-16T11:08:35.330592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:04:46.888158+0000","scrub_duration":0.0034591069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":643,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:13.636514+0000","last_change":"2024-09-16T11:08:13.636643+0000","last_active":"2024-09-16T11:08:13.636514+0000","last_peered":"2024-09-16T11:08:13.636514+0000","last_clean":"2024-09-16T11:08:13.636514+0000","last_became_active":"2024-09-16T10:51:10.894177+0000","last_became_peered":"2024-09-16T10:51:10.894177+0000","last_unstale":"2024-09-16T11:08:13.636514+0000","last_undegraded":"2024-09-16T11:08:13.636514+0000","last_fullsized":"2024-09-16T11:08:13.636514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:13.636480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:13.636480+0000","last_clean_scrub_stamp":"2024-09-16T11:08:13.636480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:26.327719+0000","scrub_duration":0.0030393299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":648,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:36.339463+0000","last_change":"2024-09-16T11:08:36.339647+0000","last_active":"2024-09-16T11:08:36.339463+0000","last_peered":"2024-09-16T11:08:36.339463+0000","last_clean":"2024-09-16T11:08:36.339463+0000","last_became_active":"2024-09-16T10:51:10.829361+0000","last_became_peered":"2024-09-16T10:51:10.829361+0000","last_unstale":"2024-09-16T11:08:36.339463+0000","last_undegraded":"2024-09-16T11:08:36.339463+0000","last_fullsized":"2024-09-16T11:08:36.339463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:36.339413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:36.339413+0000","last_clean_scrub_stamp":"2024-09-16T11:08:36.339413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:41:34.148310+0000","scrub_duration":0.0030901359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":645,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:39.650227+0000","last_change":"2024-09-16T11:08:39.650425+0000","last_active":"2024-09-16T11:08:39.650227+0000","last_peered":"2024-09-16T11:08:39.650227+0000","last_clean":"2024-09-16T11:08:39.650227+0000","last_became_active":"2024-09-16T10:51:10.842182+0000","last_became_peered":"2024-09-16T10:51:10.842182+0000","last_unstale":"2024-09-16T11:08:39.650227+0000","last_undegraded":"2024-09-16T11:08:39.650227+0000","last_fullsized":"2024-09-16T11:08:39.650227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:39.650182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:39.650182+0000","last_clean_scrub_stamp":"2024-09-16T11:08:39.650182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:12:23.507721+0000","scrub_duration":0.0035062230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"694'8","reported_seq":673,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.366157+0000","last_change":"2024-09-16T11:08:19.366353+0000","last_active":"2024-09-16T11:08:19.366157+0000","last_peered":"2024-09-16T11:08:19.366157+0000","last_clean":"2024-09-16T11:08:19.366157+0000","last_became_active":"2024-09-16T10:51:10.808073+0000","last_became_peered":"2024-09-16T10:51:10.808073+0000","last_unstale":"2024-09-16T11:08:19.366157+0000","last_undegraded":"2024-09-16T11:08:19.366157+0000","last_fullsized":"2024-09-16T11:08:19.366157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"694'8","last_scrub_stamp":"2024-09-16T11:08:19.366108+0000","last_deep_scrub":"694'8","last_deep_scrub_stamp":"2024-09-16T11:08:19.366108+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.366108+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:09:21.199912+0000","scrub_duration":0.0063593720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":676,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:20.422945+0000","last_change":"2024-09-16T11:08:20.423148+0000","last_active":"2024-09-16T11:08:20.422945+0000","last_peered":"2024-09-16T11:08:20.422945+0000","last_clean":"2024-09-16T11:08:20.422945+0000","last_became_active":"2024-09-16T10:51:10.866314+0000","last_became_peered":"2024-09-16T10:51:10.866314+0000","last_unstale":"2024-09-16T11:08:20.422945+0000","last_undegraded":"2024-09-16T11:08:20.422945+0000","last_fullsized":"2024-09-16T11:08:20.422945+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:20.422897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:20.422897+0000","last_clean_scrub_stamp":"2024-09-16T11:08:20.422897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:10:28.150274+0000","scrub_duration":0.0029436589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":650,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-09-16T11:08:19.322629+0000","last_change":"2024-09-16T11:08:19.322913+0000","last_active":"2024-09-16T11:08:19.322629+0000","last_peered":"2024-09-16T11:08:19.322629+0000","last_clean":"2024-09-16T11:08:19.322629+0000","last_became_active":"2024-09-16T10:51:10.814108+0000","last_became_peered":"2024-09-16T10:51:10.814108+0000","last_unstale":"2024-09-16T11:08:19.322629+0000","last_undegraded":"2024-09-16T11:08:19.322629+0000","last_fullsized":"2024-09-16T11:08:19.322629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:08:19.322543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:08:19.322543+0000","last_clean_scrub_stamp":"2024-09-16T11:08:19.322543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:01:31.817285+0000","scrub_duration":0.0033373700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640489,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80044,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291796,"statfs":{"total":96636764160,"available":96554799104,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.49199999999999999,"15min":0.36799999999999999},"max":{"1min":0.99399999999999999,"5min":1.2789999999999999,"15min":1.9390000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.77000000000000002,"15min":0.75700000000000001},"min":{"1min":0.57599999999999996,"5min":0.53300000000000003,"15min":0.42799999999999999},"max":{"1min":0.91000000000000003,"5min":1.123,"15min":1.984},"last":0.82499999999999996}]},{"osd":1,"last update":"Mon Sep 16 11:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.51200000000000001,"5min":0.49299999999999999,"15min":0.39400000000000002},"max":{"1min":0.96899999999999997,"5min":1.2789999999999999,"15min":1.3},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86499999999999999,"15min":0.89100000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.53000000000000003},"max":{"1min":1.0329999999999999,"5min":1.2470000000000001,"15min":2.085},"last":0.76300000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.77100000000000002,"15min":0.78200000000000003},"min":{"1min":0.624,"5min":0.57199999999999995,"15min":0.45400000000000001},"max":{"1min":0.95899999999999996,"5min":1.103,"15min":1.1779999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80200000000000005,"15min":0.82299999999999995},"min":{"1min":0.53900000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91600000000000004,"5min":1.0620000000000001,"15min":1.22},"last":0.69799999999999995}]},{"osd":3,"last update":"Mon Sep 16 11:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67100000000000004,"15min":0.72499999999999998},"min":{"1min":0.502,"5min":0.42199999999999999,"15min":0.33200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0780000000000001,"15min":1.385},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.82899999999999996,"15min":0.79700000000000004},"min":{"1min":0.64400000000000002,"5min":0.44,"15min":0.44},"max":{"1min":1.0609999999999999,"5min":1.3029999999999999,"15min":1.365},"last":0.80300000000000005}]},{"osd":4,"last update":"Mon Sep 16 11:08:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.56100000000000005,"5min":0.54000000000000004,"15min":0.39400000000000002},"max":{"1min":0.88100000000000001,"5min":1.6759999999999999,"15min":1.6759999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.94899999999999995,"15min":0.93100000000000005},"min":{"1min":0.80900000000000005,"5min":0.76400000000000001,"15min":0.499},"max":{"1min":1.0329999999999999,"5min":1.716,"15min":1.716},"last":0.88300000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640485,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80036,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291804,"statfs":{"total":96636764160,"available":96554807296,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77199813},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78100000000000003,"15min":0.82899999999999996},"min":{"1min":0.59499999999999997,"5min":0.53100000000000003,"15min":0.373},"max":{"1min":0.92300000000000004,"5min":1.169,"15min":5.351},"last":0.627},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.80400000000000005,"15min":0.78400000000000003},"min":{"1min":0.65600000000000003,"5min":0.51900000000000002,"15min":0.33900000000000002},"max":{"1min":0.97399999999999998,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.79900000000000004}]},{"osd":1,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.89300000000000002,"15min":0.91800000000000004},"min":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.318},"max":{"1min":1.0149999999999999,"5min":1.268,"15min":10.134},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.90600000000000003,"15min":0.91200000000000003},"min":{"1min":0.76200000000000001,"5min":0.58599999999999997,"15min":0.39900000000000002},"max":{"1min":1.044,"5min":1.3069999999999999,"15min":3.4369999999999998},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82299999999999995,"15min":0.89200000000000002},"min":{"1min":0.67300000000000004,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.94399999999999995,"5min":1.202,"15min":7.9119999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.68500000000000005,"5min":0.623,"15min":0.46899999999999997},"max":{"1min":1.014,"5min":1.28,"15min":1.3859999999999999},"last":0.878}]},{"osd":3,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.73599999999999999},"min":{"1min":0.52500000000000002,"5min":0.498,"15min":0.41899999999999998},"max":{"1min":0.84699999999999998,"5min":1.2010000000000001,"15min":8.5129999999999999},"last":0.749},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.77700000000000002},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.44500000000000001},"max":{"1min":1.0609999999999999,"5min":1.268,"15min":5.0579999999999998},"last":0.73399999999999999}]},{"osd":5,"last update":"Mon Sep 16 11:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.76600000000000001},"min":{"1min":0.54500000000000004,"5min":0.54000000000000004,"15min":0.372},"max":{"1min":0.91600000000000004,"5min":1.153,"15min":3.7770000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77200000000000002,"15min":0.83199999999999996},"min":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.42099999999999999},"max":{"1min":0.89900000000000002,"5min":1.111,"15min":8.5559999999999992},"last":0.84399999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640485,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76460,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94295380,"statfs":{"total":96636764160,"available":96558469120,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73005512},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77600000000000002,"15min":0.78300000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.45100000000000001},"max":{"1min":0.92200000000000004,"5min":1.1359999999999999,"15min":1.3720000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75800000000000001,"15min":0.78000000000000003},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.47399999999999998},"max":{"1min":0.995,"5min":1.0609999999999999,"15min":1.609},"last":0.77400000000000002}]},{"osd":1,"last update":"Mon Sep 16 11:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.87,"15min":0.85099999999999998},"min":{"1min":0.56699999999999995,"5min":0.56100000000000005,"15min":0.47599999999999998},"max":{"1min":1.0740000000000001,"5min":1.4430000000000001,"15min":1.821},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.94499999999999995,"15min":0.92900000000000005},"min":{"1min":0.73499999999999999,"5min":0.64500000000000002,"15min":0.499},"max":{"1min":1.079,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.82999999999999996}]},{"osd":2,"last update":"Mon Sep 16 11:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.80800000000000005,"15min":0.79300000000000004},"min":{"1min":0.65400000000000003,"5min":0.53900000000000003,"15min":0.48999999999999999},"max":{"1min":1.054,"5min":1.2290000000000001,"15min":1.581},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86199999999999999,"15min":0.84999999999999998},"min":{"1min":0.67200000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0629999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.89700000000000002}]},{"osd":4,"last update":"Mon Sep 16 11:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.54200000000000004,"5min":0.42999999999999999,"15min":0.38800000000000001},"max":{"1min":0.93999999999999995,"5min":0.999,"15min":3.4710000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.51600000000000001,"5min":0.47999999999999998,"15min":0.42299999999999999},"max":{"1min":1.0129999999999999,"5min":1.359,"15min":3.3559999999999999},"last":0.73099999999999998}]},{"osd":5,"last update":"Mon Sep 16 11:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.754,"15min":0.77000000000000002},"min":{"1min":0.52700000000000002,"5min":0.46300000000000002,"15min":0.32200000000000001},"max":{"1min":0.97099999999999997,"5min":1.1870000000000001,"15min":1.353},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.81999999999999995,"15min":0.84499999999999997},"min":{"1min":0.67500000000000004,"5min":0.52200000000000002,"15min":0.42699999999999999},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":2.625},"last":0.86799999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640485,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71844,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94299996,"statfs":{"total":96636764160,"available":96563195904,"internally_reserved":0,"allocated":4747264,"data_stored":1984718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.38900000000000001},"max":{"1min":0.70599999999999996,"5min":0.92800000000000005,"15min":1.0920000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70599999999999996,"15min":0.70899999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.39500000000000002},"max":{"1min":0.85799999999999998,"5min":1.0700000000000001,"15min":1.2390000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.438,"15min":0.36799999999999999},"max":{"1min":0.66200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.76200000000000001,"15min":0.754},"min":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.45500000000000002},"max":{"1min":0.88400000000000001,"5min":1.155,"15min":1.155},"last":0.90800000000000003}]},{"osd":3,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84199999999999997,"15min":0.85299999999999998},"min":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.115,"15min":1.9390000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.89700000000000002,"15min":0.89500000000000002},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.60099999999999998},"max":{"1min":0.99299999999999999,"5min":1.341,"15min":1.774},"last":0.94199999999999995}]},{"osd":4,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.86699999999999999,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.59499999999999997},"max":{"1min":1.0600000000000001,"5min":1.202,"15min":1.8839999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89000000000000001,"15min":0.876},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.59899999999999998},"max":{"1min":1.0249999999999999,"5min":1.101,"15min":1.7090000000000001},"last":0.77100000000000002}]},{"osd":5,"last update":"Mon Sep 16 11:08:29 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88800000000000001,"15min":0.88},"min":{"1min":0.70299999999999996,"5min":0.63800000000000001,"15min":0.59399999999999997},"max":{"1min":1.0640000000000001,"5min":1.397,"15min":3.7989999999999999},"last":0.873},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.94799999999999995},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.57799999999999996},"max":{"1min":0.96199999999999997,"5min":1.1950000000000001,"15min":3.2799999999999998},"last":0.80800000000000005}]}]},{"osd":1,"up_from":11,"seq":47244640486,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67756,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304084,"statfs":{"total":96636764160,"available":96567382016,"internally_reserved":0,"allocated":4755456,"data_stored":1984742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.19800000000000001},"max":{"1min":0.72099999999999997,"5min":1.0209999999999999,"15min":2.7469999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71499999999999997,"15min":0.72599999999999998},"min":{"1min":0.52900000000000003,"5min":0.39000000000000001,"15min":0.29199999999999998},"max":{"1min":0.96599999999999997,"5min":0.996,"15min":1.845},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon Sep 16 11:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64000000000000001,"15min":0.66400000000000003},"min":{"1min":0.51200000000000001,"5min":0.38,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":1.034,"15min":4.9260000000000002},"last":0.628},{"interface":"front","average":{"1min":0.626,"5min":0.629,"15min":0.67100000000000004},"min":{"1min":0.47299999999999998,"5min":0.42899999999999999,"15min":0.38300000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":2.8370000000000002},"last":0.73399999999999999}]},{"osd":3,"last update":"Mon Sep 16 11:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.88600000000000001,"15min":0.89500000000000002},"min":{"1min":0.64900000000000002,"5min":0.59899999999999998,"15min":0.32700000000000001},"max":{"1min":1.0580000000000001,"5min":1.2989999999999999,"15min":4.9740000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.94299999999999995,"15min":0.97699999999999998},"min":{"1min":0.753,"5min":0.71599999999999997,"15min":0.35499999999999998},"max":{"1min":1.038,"5min":1.3420000000000001,"15min":5.0129999999999999},"last":0.98599999999999999}]},{"osd":4,"last update":"Mon Sep 16 11:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.876,"15min":0.89700000000000002},"min":{"1min":0.69299999999999995,"5min":0.64400000000000002,"15min":0.47899999999999998},"max":{"1min":0.94699999999999995,"5min":1.2729999999999999,"15min":2.802},"last":1.1970000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.66100000000000003,"5min":0.53400000000000003,"15min":0.46500000000000002},"max":{"1min":1.038,"5min":1.3049999999999999,"15min":2.8580000000000001},"last":1.3089999999999999}]},{"osd":5,"last update":"Mon Sep 16 11:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85399999999999998,"15min":0.90900000000000003},"min":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.45100000000000001},"max":{"1min":1.012,"5min":1.3879999999999999,"15min":5.0890000000000004},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88500000000000001,"15min":0.90300000000000002},"min":{"1min":0.63700000000000001,"5min":0.57799999999999996,"15min":0.52900000000000003},"max":{"1min":1.075,"5min":1.2230000000000001,"15min":2.4630000000000001},"last":0.82499999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640487,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80556,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94291284,"statfs":{"total":96636764160,"available":96554274816,"internally_reserved":0,"allocated":5279744,"data_stored":2513646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54800000000000004,"15min":0.58099999999999996},"min":{"1min":0.432,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77400000000000002,"5min":0.86899999999999999,"15min":2.6469999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.621,"15min":0.63100000000000001},"min":{"1min":0.48999999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":1.079,"15min":2.3479999999999999},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.623,"15min":0.65300000000000002},"min":{"1min":0.501,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.80300000000000005,"5min":0.96099999999999997,"15min":1.327},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.54600000000000004,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.0609999999999999,"5min":1.228,"15min":4.1319999999999997},"last":0.70699999999999996}]},{"osd":3,"last update":"Mon Sep 16 11:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.86399999999999999,"15min":0.90400000000000003},"min":{"1min":0.72599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0800000000000001,"5min":1.1599999999999999,"15min":7.5110000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.85699999999999998,"15min":0.90200000000000002},"min":{"1min":0.66200000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91800000000000004,"5min":1.2549999999999999,"15min":8.5020000000000007},"last":0.77400000000000002}]},{"osd":4,"last update":"Mon Sep 16 11:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89100000000000001,"15min":0.92200000000000004},"min":{"1min":0.752,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0960000000000001,"5min":1.1919999999999999,"15min":6.2210000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86399999999999999,"15min":0.89500000000000002},"min":{"1min":0.72199999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0329999999999999,"5min":1.302,"15min":3.1440000000000001},"last":1.028}]},{"osd":5,"last update":"Mon Sep 16 11:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.91300000000000003,"15min":0.89600000000000002},"min":{"1min":0.754,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.179,"5min":1.3300000000000001,"15min":3.5070000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.876,"15min":0.93999999999999995},"min":{"1min":0.65100000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.1539999999999999,"5min":1.355,"15min":5.7190000000000003},"last":0.88400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T11:08:56.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T11:08:56.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: get_monmap_and_config 2024-09-16T11:08:56.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: build_initial_monmap 2024-09-16T11:08:56.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: monmap: 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T11:08:56.210247+0000 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T11:08:56.210247+0000 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:0: v1:172.21.15.5:6789/0 mon.noname-a-legacy 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-b 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:3: v2:172.21.15.5:3300/0 mon.noname-a 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T11:08:56.213 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 Processor -- start 2024-09-16T11:08:56.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 -- start start 2024-09-16T11:08:56.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: init 2024-09-16T11:08:56.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:56.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f352009a6f0 addr v1:172.21.15.143:6789/0 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 0x7f352009eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f352009e320 addr v2:172.21.15.143:3300/0 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f3520136f70 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35200a0300 con 0x7f352009a6f0 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f352009f160 con 0x7f352009e320 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:56.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352009be40 con 0x7f3520136f70 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.208+0000 7f351ffff700 10 monclient(hunting): authenticate will time out at 1994.787830s 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60980/0 (socket says 172.21.15.5:60980) 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 -- 172.21.15.5:0/2681093936 learned_addr learned my addr 172.21.15.5:0/2681093936 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): get_auth_request con 0x7f3520136f70 auth_method 0 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 1 --2- 172.21.15.5:0/2681093936 >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 0x7f352009eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): get_auth_request con 0x7f352009e320 auth_method 0 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:56.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient(hunting): handle_auth_done global_id 114218 payload 274 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 -- 172.21.15.5:0/2681093936 >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 msgr2=0x7f352009eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 --2- 172.21.15.5:0/2681093936 >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 0x7f352009eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 -- 172.21.15.5:0/2681093936 >> v1:172.21.15.143:6789/0 conn(0x7f352009a6f0 legacy=0x7f352009dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient: _finish_hunting 0 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 monclient: found mon.noname-b 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:56.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 -- 172.21.15.5:0/2681093936 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352009c120 con 0x7f3520136f70 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f35167fc700 1 --2- 172.21.15.5:0/2681093936 >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 0x7f352009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351f7fe700 1 --2- 172.21.15.5:0/2681093936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3518009e50 tx=0x7f351800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da75e6f3fb470f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 1 -- 172.21.15.5:0/2681093936 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351800e070 con 0x7f3520136f70 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: dump: 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:56.224 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: _finish_auth 0 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: _check_auth_tickets 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 5 monclient: authenticate success, global_id 114218 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 1 -- 172.21.15.5:0/2681093936 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351800b910 con 0x7f3520136f70 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 10 monclient: get_monmap_and_config success 2024-09-16T11:08:56.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 4 set_mon_vals no callback set 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 1 -- 172.21.15.5:0/2681093936 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351800bcc0 con 0x7f3520136f70 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351e7fc700 10 monclient: dump: 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:56.226 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 10 monclient: shutdown 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 -- 172.21.15.5:0/2681093936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 msgr2=0x7f3520139360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 --2- 172.21.15.5:0/2681093936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3518009e50 tx=0x7f351800b330 comp rx=0 tx=0).stop 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 -- 172.21.15.5:0/2681093936 shutdown_connections 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 --2- 172.21.15.5:0/2681093936 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f3520136f70 0x7f3520139360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 --2- 172.21.15.5:0/2681093936 >> v2:172.21.15.143:3300/0 conn(0x7f352009e320 0x7f352009eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 -- 172.21.15.5:0/2681093936 >> 172.21.15.5:0/2681093936 conn(0x7f35200a4450 msgr2=0x7f35200a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 -- 172.21.15.5:0/2681093936 shutdown_connections 2024-09-16T11:08:56.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.212+0000 7f351ffff700 1 -- 172.21.15.5:0/2681093936 wait complete. 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: build_initial_monmap 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: monmap: 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 0 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 0.000000 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:created 0.000000 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 0 (unknown) 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 1 2024-09-16T11:08:56.232 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.noname-a 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.noname-c 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 librados: starting msgr at 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 librados: starting objecter 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 Processor -- start 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- start start 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 librados: setting wanted keys 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 librados: calling monclient init 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: init 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _reopen_session rank -1 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f352009a6f0 addr [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f352009e320 0x7f3520135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f352009e320 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3520136f70 0x7f3520135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f3520136f70 addr [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352006de80 con 0x7f3520136f70 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352013b750 con 0x7f352009e320 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352009b220 con 0x7f352009a6f0 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): _renew_subs 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient(hunting): authenticate will time out at 1994.793936s 2024-09-16T11:08:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 --2- >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3301/0 says I am v2:172.21.15.5:60982/0 (socket says 172.21.15.5:60982) 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 -- 172.21.15.5:0/170969396 learned_addr learned my addr 172.21.15.5:0/170969396 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): get_auth_request con 0x7f352009a6f0 auth_method 0 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3520136f70 0x7f3520135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): get_auth_request con 0x7f3520136f70 auth_method 0 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f352009e320 0x7f3520135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): get_auth_request con 0x7f352009e320 auth_method 0 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f35167fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient(hunting): handle_auth_done global_id 114221 payload 824 2024-09-16T11:08:56.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 -- 172.21.15.5:0/170969396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f352009e320 msgr2=0x7f3520135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f352009e320 0x7f3520135350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 -- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3520136f70 msgr2=0x7f3520135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3520136f70 0x7f3520135890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient: _finish_hunting 0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 monclient: found mon.noname-c 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 -- 172.21.15.5:0/170969396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080094b0 con 0x7f352009a6f0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351effd700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f351000c440 tx=0x7f351000c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 client.?.objecter ms_handle_connect 0x7f352009a6f0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 client.?.objecter resend_mon_ops 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100052d0 con 0x7f352009a6f0 2024-09-16T11:08:56.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: dump: 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: _finish_auth 0 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: _check_auth_tickets 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510005430 con 0x7f352009a6f0 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 5 monclient: authenticate success, global_id 114221 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _renew_subs 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3518009b30 con 0x7f352009a6f0 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35100055a0 con 0x7f352009a6f0 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f3517fff700 10 monclient: dump: 2024-09-16T11:08:56.239 INFO:teuthology.orchestra.run.smithi005.stderr:epoch 1 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:fsid 0ea84356-fa95-422b-b956-38c4a6b7c43e 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:last_changed 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:created 2024-09-16T10:49:21.278459+0000 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:min_mon_release 18 (reef) 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:election_strategy: 3 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:2: [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon.c 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351dffb700 4 set_mon_vals no callback set 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 client.114221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _renew_subs 2024-09-16T11:08:56.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352006d570 con 0x7f352009a6f0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.216+0000 7f351ffff700 1 librados: init done 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 4 ==== mgrmap(e 5) v1 ==== 43272+0+0 (secure 0 0 0) 0x7f35100036a0 con 0x7f352009a6f0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f351ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f351ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f351ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 1 --2- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 0x7f3500039420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352009eaa0 con 0x7f352009a6f0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 5 ==== osd_map(694..694 src has 395..694) v4 ==== 4615+0+0 (secure 0 0 0) 0x7f3510046c70 con 0x7f352009a6f0 2024-09-16T11:08:56.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 10 client.114221.objecter ms_dispatch 0x7f3520000fd0 osd_map(694..694 src has 395..694) v4 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 3 client.114221.objecter handle_osd_map got epochs [694,694] > 0 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 3 client.114221.objecter handle_osd_map decoding full epoch 694 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 20 client.114221.objecter dump_active .. 0 homeless 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f35167fc700 1 --2- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 0x7f3500039420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f35167fc700 10 monclient: get_auth_request con 0x7f3500036f60 auth_method 0 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f3517fff700 10 client.114221.objecter ms_handle_connect 0x7f3500036f60 2024-09-16T11:08:56.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.220+0000 7f35167fc700 1 --2- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 0x7f3500039420 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7f3520071b40 tx=0x7f3508009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T11:08:56.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.228+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f35100108e0 con 0x7f352009a6f0 2024-09-16T11:08:56.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.228+0000 7f3517fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T11:08:56.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.228+0000 7f3517fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T11:08:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.400+0000 7f351ffff700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T11:08:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.400+0000 7f351ffff700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T11:08:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.400+0000 7f351ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.5:3301/0 2024-09-16T11:08:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.400+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 --> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f352013b350 con 0x7f352009a6f0 2024-09-16T11:08:56.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.408+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3510003e70 con 0x7f352009a6f0 2024-09-16T11:08:56.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.408+0000 7f3517fff700 10 monclient: handle_config config(1 keys) v1 2024-09-16T11:08:56.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.408+0000 7f351dffb700 4 set_mon_vals no callback set 2024-09-16T11:08:56.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.408+0000 7f351dffb700 10 set_mon_vals mon_health_to_clog = false 2024-09-16T11:08:56.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.412+0000 7f3517fff700 1 -- 172.21.15.5:0/170969396 <== mon.2 v2:172.21.15.5:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) v1 ==== 123+0+0 (secure 0 0 0) 0x7f3510010700 con 0x7f352009a6f0 2024-09-16T11:08:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.412+0000 7f3517fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T11:08:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.412+0000 7f3517fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T11:08:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 10 librados: watch_flush enter 2024-09-16T11:08:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 10 librados: watch_flush exit 2024-09-16T11:08:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 20 client.114221.objecter shutdown clearing up homeless session... 2024-09-16T11:08:56.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 10 client.114221.objecter successfully canceled tick 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 msgr2=0x7f3500039420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 0x7f3500039420 secure :-1 s=READY pgs=3931 cs=0 l=1 rev1=1 crypto rx=0x7f3520071b40 tx=0x7f3508009240 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 10 monclient: shutdown 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 msgr2=0x7f3520134e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f351000c440 tx=0x7f351000c750 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 shutdown_connections 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> v2:172.21.15.143:6812/1098205034 conn(0x7f3500036f60 0x7f3500039420 unknown :-1 s=CLOSED pgs=3931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3520136f70 0x7f3520135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f352009e320 0x7f3520135350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 --2- 172.21.15.5:0/170969396 >> [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] conn(0x7f352009a6f0 0x7f3520134e10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T11:08:56.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 >> 172.21.15.5:0/170969396 conn(0x7f35200a4450 msgr2=0x7f3520130010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T11:08:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 shutdown_connections 2024-09-16T11:08:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 -- 172.21.15.5:0/170969396 wait complete. 2024-09-16T11:08:56.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-16T11:08:56.416+0000 7f351ffff700 1 librados: shutdown 2024-09-16T11:08:56.431 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T11:08:56.431 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T11:08:56.431 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T11:08:56.432 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:01.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-16T11:09:02.533 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T11:09:02.534 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T11:09:02.534 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:02.624 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T11:09:02.624 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T11:09:02.624 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:02.720 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T11:09:02.720 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T11:09:02.720 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:07.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-16T11:09:07.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-16T11:09:08.822 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T11:09:08.822 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T11:09:08.822 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:08.910 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T11:09:08.910 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T11:09:08.910 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:09.002 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T11:09:09.002 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T11:09:09.003 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T11:09:09.003 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:09.033 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T11:09:09.033 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T11:09:09.033 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T11:09:09.033 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:09.078 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T11:09:09.078 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T11:09:09.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:09.123 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T11:09:09.124 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T11:09:09.124 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:09:09.170 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T11:09:09.171 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T11:09:09.171 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-16T11:09:09.192 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T11:09:09.193 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T11:09:09.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T11:09:09.473 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T11:09:09.681 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T11:09:09.681 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T11:09:09.855 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T11:09:09.855 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T11:09:10.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T11:09:10.073 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T11:09:10.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T11:09:10.297 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T11:09:10.493 INFO:tasks.ceph:Archiving mon data... 2024-09-16T11:09:10.495 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/data/mon.a.tgz 2024-09-16T11:09:10.495 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-16T11:09:10.502 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.m0V4ANfxje 2024-09-16T11:09:10.502 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.m0V4ANfxje 2024-09-16T11:09:10.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.m0V4ANfxje 2024-09-16T11:09:10.660 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.m0V4ANfxje is 629KB 2024-09-16T11:09:10.728 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.m0V4ANfxje 2024-09-16T11:09:10.735 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/data/mon.c.tgz 2024-09-16T11:09:10.735 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-16T11:09:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.q4KhkkPYHB 2024-09-16T11:09:10.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.q4KhkkPYHB 2024-09-16T11:09:10.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.q4KhkkPYHB 2024-09-16T11:09:10.940 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.q4KhkkPYHB is 628KB 2024-09-16T11:09:11.008 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.q4KhkkPYHB 2024-09-16T11:09:11.015 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/data/mon.b.tgz 2024-09-16T11:09:11.015 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-16T11:09:11.021 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.KqJL3M3j9P 2024-09-16T11:09:11.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KqJL3M3j9P 2024-09-16T11:09:11.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.KqJL3M3j9P 2024-09-16T11:09:11.214 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.KqJL3M3j9P is 628KB 2024-09-16T11:09:11.283 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.KqJL3M3j9P 2024-09-16T11:09:11.290 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T11:09:11.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:09:11.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:09:11.308 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T11:09:11.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:09:11.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:09:11.333 INFO:teuthology.orchestra.run.smithi005.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:09:11.335 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:09:11.341 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T11:09:11.342 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi005/crash 2024-09-16T11:09:11.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:09:11.359 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi143/crash 2024-09-16T11:09:11.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:09:11.400 INFO:tasks.ceph:Compressing logs... 2024-09-16T11:09:11.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:09:11.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:09:12.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-16T11:09:33.932 INFO:tasks.ceph:Archiving logs... 2024-09-16T11:09:33.933 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi005/log 2024-09-16T11:09:33.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:09:37.037 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi143/log 2024-09-16T11:09:37.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:09:39.832 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T11:09:39.841 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T11:09:39.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:09:39.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:09:39.912 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T11:09:39.922 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T11:09:39.922 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:09:39.925 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout:============================================================================== 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout:*hv01.front.sepi 67.205.162.81 3 u 20 64 377 0.097 1.175 0.297 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout:+hv02.front.sepi 63.231.80.2 3 u 26 64 377 0.069 -1.136 0.258 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout:+hv03.front.sepi 74.6.168.72 3 u 24 64 377 0.100 1.466 0.342 2024-09-16T11:09:39.944 INFO:teuthology.orchestra.run.smithi005.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-16T11:09:39.951 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T11:09:39.951 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-09-16T11:09:39.951 INFO:teuthology.orchestra.run.smithi143.stdout:*hv01.front.sepi 67.205.162.81 3 u 23 64 377 0.101 -1.191 0.186 2024-09-16T11:09:39.951 INFO:teuthology.orchestra.run.smithi143.stdout:+hv02.front.sepi 63.231.80.2 3 u 16 64 377 0.099 -3.496 0.297 2024-09-16T11:09:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:+hv03.front.sepi 74.6.168.72 3 u 20 64 377 0.101 -1.304 0.309 2024-09-16T11:09:39.952 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-16T11:09:39.952 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T11:09:40.024 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T11:09:40.024 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T11:09:40.075 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T11:09:40.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T11:09:40.134 INFO:teuthology.task.internal:Duration was 1684.601112 seconds 2024-09-16T11:09:40.134 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T11:09:40.169 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T11:09:40.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:09:40.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:09:40.228 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T11:09:40.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-09-16T11:09:40.229 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:09:40.241 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T11:09:40.242 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:09:40.257 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T11:09:40.257 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:09:40.259 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:09:40.339 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T11:09:40.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:09:40.344 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:09:40.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T11:09:40.430 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T11:09:40.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:09:40.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:09:40.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T11:09:40.475 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:40.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:40.505 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-09-16T11:09:40.525 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-09-16T11:09:40.539 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:09:40.546 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:09:40.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T11:09:40.564 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T11:09:40.565 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi005 2024-09-16T11:09:40.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:09:40.596 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906688/remote/smithi143 2024-09-16T11:09:40.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:09:40.623 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T11:09:40.623 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:09:40.635 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:09:40.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T11:09:40.680 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T11:09:40.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T11:09:40.736 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T11:09:40.736 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:09:40.738 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:09:40.743 INFO:teuthology.orchestra.run.smithi005.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:09 /home/ubuntu/cephtest 2024-09-16T11:09:40.746 INFO:teuthology.orchestra.run.smithi143.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:09 /home/ubuntu/cephtest 2024-09-16T11:09:40.749 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T11:09:40.781 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T11:09:40.865 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/force-sync-many workloads/rados_api_tests} duration: 1684.6011123657227 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T11:09:40.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:09:40.939 INFO:teuthology.run:pass